November 2019 - ceph-users

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

add debian buster stable support for ceph-deploy
by Jelle de Jong 07 Sep '20

07 Sep '20

Hello everybody, Can somebody add support for Debian buster and ceph-deploy: https://tracker.ceph.com/issues/42870 Highly appreciated, Regards, Jelle de Jong

6 8

RBD logs
by 陈旭 04 Jun '20

04 Jun '20

Hi guys, I deploy an efk cluster and use ceph as block storage in kubernetes, but RBD write iops sometimes becomes zero and last for a few minutes. I want to check logs about RBD so I add some config to ceph.conf and restart ceph. Here is my ceph.conf: [global] fsid = 53f4e1d5-32ce-4e9c-bf36-f6b54b009962 mon_initial_members = db-16-4-hzxs, db-16-5-hzxs, db-16-6-hzxs mon_host = 10.25.16.4,10.25.16.5,10.25.16.6 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd pool default size = 3 [client] debug rbd = 20 debug rbd mirror = 20 debug rbd replay = 20 log file = /var/log/ceph/client_rbd.log I can not get any logs in /var/log/ceph/client_rbd.log. I also try to execute 'ceph daemon osd.* config set debug_rbd 20’ and there is also no related logs in ceph-osd.log. How can I get useful logs about this question or How can I analyze this problem? Look forward to your reply. Thanks ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 声明：此邮件可能包含依图公司保密或特权信息，并且仅应发送至有权接收该邮件的收件人。如果您无权收取该邮件，您应当立即删除该邮件并通知发件人，您并被禁止传播、分发或复制此邮件以及附件。对于此邮件可能携带的病毒引起的任何损害，本公司不承担任何责任。此外，本公司不保证已正确和完整地传输此信息，也不接受任何延迟收件的赔偿责任。 ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// Notice: This email may contain confidential or privileged information of Yitu and was sent solely to the intended recipients. If you are unauthorized to receive this email, you should delete the email and contact the sender immediately. Any unauthorized disclosing, distribution, or copying of this email and attachment thereto is prohibited. Yitu does not accept any liability for any loss caused by possibly viruses in this email. E-mail transmission cannot be guaranteed to be secure or error-free and Yitu is not responsible for any delayed transmission.

2 1

rbd image naming convention
by Palanisamy 29 May '20

29 May '20

Hello Team, We've integrated Ceph cluster storage with Kubernetes and provisioning volumes through rbd-provisioner. When we're creating volumes from yaml files in Kubernetes, pv > pvc > mounting to pod, In kubernetes end pvc are showing as meaningful naming convention as per yaml file defined. But in ceph cluster, rbd image name is creating with dynamic uid. During troubleshooting time, this will be tedious to find exact rbd image. Please find the provisioning logs in below pasted snippet. kubectl get pods,pv,pvc NAME READY STATUS RESTARTS AGE pod/sleepypod 1/1 Running 0 4m9s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO Delete Bound default/test-dyn-pvc ceph-rbd 4m9s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/test-dyn-pvc Bound pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO ceph-rbd 4m11s *rbd-provisioner logs* I1121 10:59:15.009012 1 provision.go:132] successfully created rbd image "kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2" I1121 10:59:15.009092 1 controller.go:1087] provision "default/test-dyn-pvc" class "ceph-rbd": volume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" provisioned I1121 10:59:15.009138 1 controller.go:1101] provision "default/test-dyn-pvc" class "ceph-rbd": trying to save persistentvvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" I1121 10:59:15.020418 1 controller.go:1108] provision "default/test-dyn-pvc" class "ceph-rbd": persistentvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" saved I1121 10:59:15.020476 1 controller.go:1149] provision "default/test-dyn-pvc" class "ceph-rbd": succeeded I1121 10:59:15.020802 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test-dyn-pvc", UID:"cd37d2d6-cecc-4a05-9736-c8d80abde7f5", APIVersion:"v1", ResourceVersion:"24545639", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 *rbd image details in Ceph cluster end* rbd -p kube ls --long NAME SIZE PARENT FMT PROT LOCK kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2 1 GiB 2 is there way to setup proper naming convention for rbd image as well during kubernetes deployment itself. Kubernetes version: v1.15.5 Ceph cluster version: 14.2.2 nautilus (stable) *Best Regards,* *Palanisamy*

2 2

ceph osd set-require-min-compat-client jewel failure
by 潘东元 14 May '20

14 May '20

hi,every one, my ceph version 12.2.12，I want to set require min compat client luminous,I use command #ceph osd set-require-min-compat-client luminous but ceph report:Error EPERM: cannot set require_min_compat_client to luminous: 4 connected client(s) look like jewel (missing 0xa00000000200000); add --yes-i-really-mean-it to do it anyway [root@node-1 ~]# ceph features { "mon": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 3 } }, "osd": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 15 } }, "client": { "group": { "features": "0x40106b84a842a52", "release": "jewel", "num": 4 }, "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 168 } } } so,I run command: [root@node-1 gyt]# ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it set require_min_compat_client to luminous but now,I want to set require min compat client jewel,I use command： [root@node-1 gyt]# ceph osd set-require-min-compat-client jewel Error EPERM: osdmap current utilizes features that require luminous; cannot set require_min_compat_client below that to jewel what‘s the way we are set luminous chang to jewel？

3 2

ceph-mgr Module "zabbix" cannot send Data
by i.schmidt＠langeoog.de 07 Apr '20

07 Apr '20

Hi Folks We are using Ceph as our storage backend on our 6 Node Proxmox VM Cluster. To Monitor our systems we use Zabbix and i would like to get some Ceph Data into our Zabbix to get some alarms when something goes wrong. Ceph mgr has a module, "zabbix" that uses "zabbix-sender" to actively send data, but i cannot get the module working. It always responds with "failed to send data" The network side seems to be fine: root@vm-2:~# traceroute 192.168.15.253 traceroute to 192.168.15.253 (192.168.15.253), 30 hops max, 60 byte packets 1 192.168.15.253 (192.168.15.253) 0.411 ms 0.402 ms 0.393 ms root@vm-2:~# nmap -p 10051 192.168.15.253 Starting Nmap 7.70 ( https://nmap.org ) at 2019-09-18 08:40 CEST Nmap scan report for 192.168.15.253 Host is up (0.00026s latency). PORT STATE SERVICE 10051/tcp open zabbix-trapper MAC Address: BA:F5:30:EF:40:EF (Unknown) Nmap done: 1 IP address (1 host up) scanned in 0.61 seconds root@vm-2:~# ceph zabbix config-show {"zabbix_port": 10051, "zabbix_host": "192.168.15.253", "identifier": "VM-2", "zabbix_sender": "/usr/bin/zabbix_sender", "interval": 60} root@vm-2:~# But if i try "ceph zabbix send" i get "failed to send data to zabbix" and this show up in the systems journal: Sep 18 08:41:13 vm-2 ceph-mgr[54445]: 2019-09-18 08:41:13.272 7fe360fe4700 -1 mgr.server reply reply (1) Operation not permitted The log of ceph-mgr on that machine states: 2019-09-18 08:42:18.188 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253392]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000179"}] 2019-09-18 08:43:18.217 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253629]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000321"}] I'm guessing, this could have something to do with user rights. But i have no idea where to start to track this down. Maybe someone here has a hint? If more information is needed, i will gladly provide it. greetings Ingo

5 6

osd_pg_create causing slow requests in Nautilus
by Bryan Stillwell 03 Apr '20

03 Apr '20

We've run into a problem on our test cluster this afternoon which is running Nautilus (14.2.2). It seems that any time PGs move on the cluster (from marking an OSD down, setting the primary-affinity to 0, or by using the balancer), a large number of the OSDs in the cluster peg the CPU cores they're running on for a while which causes slow requests. From what I can tell it appears to be related to slow peering caused by osd_pg_create() taking a long time. This was seen on quite a few OSDs while waiting for peering to complete: # ceph daemon osd.3 ops { "ops": [ { "description": "osd_pg_create(e179061 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:34:46.556413", "age": 318.25234538000001, "duration": 318.25241895300002, "type_data": { "flag_point": "started", "events": [ { "time": "2019-08-27 14:34:46.556413", "event": "initiated" }, { "time": "2019-08-27 14:34:46.556413", "event": "header_read" }, { "time": "2019-08-27 14:34:46.556299", "event": "throttled" }, { "time": "2019-08-27 14:34:46.556456", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456901", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456903", "event": "wait for new map" }, { "time": "2019-08-27 14:40:01.292346", "event": "started" } ] } }, ...snip... { "description": "osd_pg_create(e179066 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:35:09.908567", "age": 294.900191001, "duration": 294.90068416899999, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-08-27 14:35:09.908567", "event": "initiated" }, { "time": "2019-08-27 14:35:09.908567", "event": "header_read" }, { "time": "2019-08-27 14:35:09.908520", "event": "throttled" }, { "time": "2019-08-27 14:35:09.908617", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456921", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456923", "event": "wait for new map" } ] } } ], "num_ops": 6 } That "wait for new map" message made us think something was getting hung up on the monitors, so we restarted them all without any luck. I'll keep investigating, but so far my google searches aren't pulling anything up so I wanted to see if anyone else is running into this? Thanks, Bryan

6 16

RGWReshardLock::lock failed to acquire lock ret=-16
by Josh Haft 12 Mar '20

12 Mar '20

Hi, Currently running Mimic 13.2.5. We had reports this morning of timeouts and failures with PUT and GET requests to our Ceph RGW cluster. I found these messages in the RGW log: RGWReshardLock::lock failed to acquire lock on bucket_name:bucket_instance ret=-16 NOTICE: resharding operation on bucket index detected, blocking block_while_resharding ERROR: bucket is still resharding, please retry Which were preceded by many of these, which I think are normal/expected. check_bucket_shards: resharding needed: stats.num_objects=6415879 shard max_objects=6400000 Our RGW cluster sits behind haproxy which notified me approx 90 seconds after the first 'resharding needed' message that no backends were available. It appears this dynamic reshard process caused the RGWs to lock up for a period of time. Roughly 2 minutes later the reshard error messages stop and operation returns to normal. Looking back through previous RGW logs, I see a similar event from about a week ago, on the same bucket. We have several buckets with shard counts exceeding 1k (this one only has 128), and much larger object counts, so clearly this isn't the first time dynamic sharding has been invoked on this cluster. Has anyone seen this? I expect it will come up again, and can turn up debugging if that'll help. Thanks for any assistance! Josh

1 1

official ceph.com buster builds?
by Chad W Seys 27 Feb '20

27 Feb '20

Hi all, Am I missing the ceph buster build built by ceph.com ? http://download.ceph.com/debian-nautilus/dists/ Should I be using the Croit supplied builds? Thanks! Chad.

2 2

ceph status reports: slow ops - this is related to long running process /usr/bin/ceph-osd
by Thomas 18 Feb '20

18 Feb '20

Hi, ceph status reports: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available 4 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 59 pgs inactive, 16 pgs peering Degraded data redundancy: 597/153910758 objects degraded (0.000%), 2 pgs degraded, 1 pg undersized Degraded data redundancy (low space): 23 pgs backfill_toofull 1 pgs not deep-scrubbed in time 4 pgs not scrubbed in time 3 pools have too many placement groups 164 slow requests are blocked > 32 sec 1082 stuck requests are blocked > 4096 sec 1490 slow ops, oldest one blocked for 19711 sec, daemons [osd,0,osd,175,osd,186,osd,5,osd,6,osd,63,osd,68,osd,9,mon,ld5505,mon,ld5506]... have slow ops. services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 5h) mgr: ld5507(active, since 5h), standbys: ld5506, ld5505 mds: pve_cephfs:0/1, 1 failed osd: 419 osds: 416 up, 416 in; 6024 remapped pgs data: pools: 6 pools, 8864 pgs objects: 51.30M objects, 196 TiB usage: 594 TiB used, 907 TiB / 1.5 PiB avail pgs: 0.666% pgs not active 597/153910758 objects degraded (0.000%) 52964415/153910758 objects misplaced (34.412%) 5954 active+remapped+backfill_wait 2786 active+clean 40 active+remapped+backfilling 35 activating 23 active+remapped+backfill_wait+backfill_toofull 16 peering 7 activating+remapped 1 activating+undersized+degraded 1 active+clean+scrubbing 1 active+recovering+degraded io: client: 3.5 KiB/s wr, 0 op/s rd, 0 op/s wr recovery: 551 MiB/s, 137 objects/s I'm concerned about the slow ops on osd.0 and osd.9. On the relevant OSD node I can see 2 relevant services running for hours: ceph 14795 1 99 09:58 ? 08:49:22 /usr/bin/ceph-osd -f --cluster ceph --id 9 --setuser ceph --setgroup ceph ceph 15394 1 99 09:58 ? 07:10:00 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph In the relevant osd log I can find similar messages: root@ld5505:~# tail -f /var/log/ceph/ceph-osd.0.log 2019-10-08 15:35:32.830 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:33.806 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:34.842 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:35.862 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) root@ld5505:~# tail -f /var/log/ceph/ceph-osd.9.log 2019-10-08 15:35:38.822 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:39.854 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:40.850 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:41.862 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) Question: How can I analyse and solve the issue with slow ops? THX

3 2

Changing failure domain
by Francois Legrand 01 Feb '20

01 Feb '20

Hi, I have a cephfs in production based on 2 pools (data+metadata). Data is in erasure coding with the profile : crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=3 m=2 plugin=jerasure technique=reed_sol_van w=8 Metadata is in replicated mode with k=3 The crush rules are as follow : [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "ec_data", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 5, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] When we installed it, everything was in the same room, but know we splitted our cluster (6 servers but soon 8) in 2 rooms. Thus we updated the crushmap by adding a room layer (with ceph osd crush add-bucket room1 room etc) and move all our servers in the tree to the correct place (ceph osd crush move server1 room=room1 etc...). Now, we would like to change the rules to set a failure domain to room instead of host (to be sure that in case of disaster in one of the rooms we will still have a copy in the other). What is the best strategy to do this ? F.

4 10

Radosgw/Objecter behaviour for homeless session
by Biswajeet Patra 23 Jan '20

23 Jan '20

Hi All, I have a query regarding objecter behaviour for homeless session. In situations when all OSDs containing copies (*let say replication 3*) of an object are down, the objecter assigns a homeless session (OSD=-1) to a client request. This request makes radosgw thread hang indefinitely as the data could never be served because all required OSDs are down. With multiple similar requests, all the radosgw threads gets exhausted and hanged indefinitely waiting for the OSDs to come up. This creates complete service unavailability as no rgw threads are present to process valid requests which could have been directed towards active PGs/OSDs. I think we should have behaviour in objecter or radosgw to terminate request and return early in case of a homeless session. Let me know your thoughts on this. Regards, Biswajeet -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 1

why osd's heartbeat partner comes from another root tree?
by opengers 08 Jan '20

08 Jan '20

Hello: According to my understanding, osd's heartbeat partners only come from those osds who assume the same pg See below(# ceph osd tree), osd.10 and osd.0-6 cannot assume the same pg, because osd.10 and osd.0-6 are from different root tree, and pg in my cluster doesn't map across root trees(# ceph osd crush rule dump). so, osd.0-6 cannot become the heartbeat partner of osd.10 But, below is the log on osd.10, It can be seen that the osd.10's heartbeat partner include osd.0/1/2/5, why? thanks for any help > # osd.10 log > 2019-11-20 09:21:50.431799 7fbb369fb700 -1 osd.10 7344 heartbeat_check: no > reply from 10.13.6.162:6806 osd.2 since back 2019-11-20 09:21:19.979712 > front 2019-11-20 09:21:19.979712 (cutoff 2019-11-20 09:21:30.431768) > 2019-11-20 13:15:59.175060 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no > reply from 10.13.6.162:6806 <http://10.13.6.152:6806> osd.2 since back > 2019-11-20 13:15:38.710424 front 2019-11-20 13:15:38.710424 (cutoff > 2019-11-20 13:15:39.175058) > 2019-11-20 13:15:59.175110 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no > reply from 10.13.6.160:6803 osd.0 since back 2019-11-20 13:15:38.710424 > front 2019-11-20 13:15:38.710424 (cutoff 2019-11-20 13:15:39.175058) > 2019-11-20 13:15:59.175118 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no > reply from 10.13.6.161:6803 osd.1 since back 2019-11-20 13:15:38.710424 > front 2019-11-20 13:15:38.710424 (cutoff 2019-11-20 13:15:39.175058) > 2019-11-21 02:52:24.656783 7fbb369fb700 -1 osd.10 7374 heartbeat_check: no > reply from 10.13.6.158:6810 osd.5 since back 2019-11-21 02:52:04.557548 > front 2019-11-21 02:52:04.557548 (cutoff 2019-11-21 02:52:04.656781) > # ceph osd tree > -17 3.29095 root ssd-storage > > -25 1.09698 rack rack-ssd-A > > -18 1.09698 host ssd-osd01 > 10 hdd 1.09698 osd.10 up 1.00000 > 1.00000 > -26 1.09698 rack rack-ssd-B > > -19 1.09698 host ssd-osd02 > 11 hdd 1.09698 osd.11 up 1.00000 > 1.00000 > -27 1.09698 rack rack-ssd-C > > -20 1.09698 host ssd-osd03 > 12 hdd 1.09698 osd.12 up 1.00000 > 1.00000 > -1 3.22256 root default > > -3 0.29300 host test-osd01 > > 0 hdd 0.29300 osd.0 up 1.00000 > 1.00000 > -5 0.29300 host test-osd02 > > 1 hdd 0.29300 osd.1 up 0.89999 > 1.00000 > -7 0.29300 host test-osd03 > > 2 hdd 0.29300 osd.2 up 0.79999 > 1.00000 > -9 0.29300 host test-osd04 > > 3 hdd 0.29300 osd.3 up 1.00000 > 1.00000 > -11 0.29300 host test-osd05 > > 4 hdd 0.29300 osd.4 up 1.00000 > 1.00000 > -13 0.29300 host test-osd06 > > 5 hdd 0.29300 osd.5 up 1.00000 > 1.00000 > -15 0.29300 host test-osd07 > > 6 hdd 0.29300 osd.6 up 1.00000 > 1.00000 # ceph osd crush rule dump > > [ > { > "rule_id": 0, > "rule_name": "replicated_rule", > "ruleset": 0, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -1, > "item_name": "default" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "host" > }, > { > "op": "emit" > } > ] > }, > { > "rule_id": 1, > "rule_name": "replicated_rule_ssd", > "ruleset": 1, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -17, > "item_name": "ssd-storage" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "rack" > }, > { > "op": "emit" > } > ] > } > ] > # some parameters > "mon_osd_down_out_interval": "600", > "mon_osd_down_out_subtree_limit": "rack", > "mds_debug_subtrees": "false", > "mon_osd_down_out_subtree_limit": "rack", > "mon_osd_reporter_subtree_level": "host",

3 3

MDS failing under load with large cache sizes
by Janek Bevendorff 07 Jan '20

07 Jan '20

Hi, I am trying to copy the contents of our storage server into a CephFS, but am experiencing stability issues with my MDSs. The CephFS sits on top of an erasure-coded pool with 5 MONs, 5 MDSs and a max_mds setting of two. My Ceph cluster version is Nautilus, the client is Mimic and uses the kernel module to mount the FS. The index of filenames to copy is about 23GB and I am using 16 parallel rsync processes over a 10G link to copy the files over to Ceph. This works perfectly for a while, but then the MDSs start reporting oversized caches (between 20 and 50GB, sometimes more) and an inode count between 1 and 4 million. Particularly the Inode count seems quite high to me. Each rsync job has 25k files to work with, so if all 16 processes open all their files at the same time, I should not exceed 400k. Even if I double this number to account for the client's page cache, I should get nowhere near that number of inodes (a sync flush takes about 1 second). Then after a few hours, my MDSs start failing with messages like this: -21> 2019-07-22 14:00:05.877 7f67eacec700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -20> 2019-07-22 14:00:05.877 7f67eacec700 0 mds.beacon.XXX Skipping beacon heartbeat to monitors (last acked 24.0042s ago); MDS internal heartbeat is not healthy! The standby nodes try to take over, but take forever to become active and will fail as well eventually. During my research, I found this related topic: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015959.html, but I tried everything in there from increasing to lowering my cache size, the number of segments etc. I also played around with the number of active MDSs and two appears to work the best, whereas one cannot keep up with the load and three seems to be the worst of all choices. Do you have any ideas how I can improve the stability of my MDS damons to handle the load properly? single 10G link is a toy and we could query the cluster with a lot more requests per second, but it's already yielding to 16 rsync processes. Thanks

4 33

Balancing PGs across OSDs
by Thomas Schneider 06 Jan '20

06 Jan '20

Hi, in this <https://ceph.io/community/the-first-telemetry-results-are-in/> blog post I find this statement: "So, in our ideal world so far (assuming equal size OSDs), every OSD now has the same number of PGs assigned." My issue is that accross all pools the number of PGs per OSD is not equal. And I conclude that this is causing very unbalanced data placement. As a matter of fact the data stored on my 1.6TB HDD in specific pool "hdb_backup" is in a range starting with osd.228 size: 1.6 usage: 52.61 reweight: 1.00000 and ending with osd.145 size: 1.6 usage: 81.11 reweight: 1.00000 This impacts the amount of data that can be stored in the cluster heavily. Ceph balancer is enabled, but this is not solving this issue. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } Therefore I would ask you for suggestions how to work on this unbalanced data distribution. I have attached pastebin for - ceph osd df sorted by usage <https://pastebin.com/QLQHjA9g> - ceph osd df tree <https://pastebin.com/SvhP2hp5> My cluster has multiple crush roots respresenting different disks. In addition I have defined multiple pools, one pool for each disk type: hdd, ssd, nvme. THX

6 23

Ceph and centos 8
by fleg＠lpnhe.in2p3.fr 17 Dec '19

17 Dec '19

Hi, We have a ceph+cephfs cluster runing nautilus version 14.2.4 We have debian buster/ubuntu bionic clients mounting cephfs in kernel mode without problems. We now want to mount cephfs from our new centos 8 clients. Unfortunately, ceph-common is needed but there are no packages available for el8 (only el7). And no way to install the el7 packages on centos 8 (missing deps). Thus, despite the fact that centos 8 have a 4.18 kernel (required to use quota, snapshots etc...), it seems impossible to mount in kernel mode (good perfs) and we still have to use the so slow fuse mode. Is it possible to workaround this problem ? Or when is it planned to provides (even as beta) the ceph packages for centos 8 ? Thanks.

4 5

v13.2.7 mimic released
by Sage Weil 16 Dec '19

16 Dec '19

This is the seventh bugfix release of the Mimic v13.2.x long term stable release series. We recommend all Mimic users upgrade. For the full release notes, see https://ceph.io/releases/v13-2-7-mimic-released/ Notable Changes MDS: - Cache trimming is now throttled. Dropping the MDS cache via the “ceph tell mds.<foo> cache drop” command or large reductions in the cache size will no longer cause service unavailability. - Behavior with recalling caps has been significantly improved to not attempt recalling too many caps at once, leading to instability. MDS with a large cache (64GB+) should be more stable. - MDS now provides a config option “mds_max_caps_per_client” (default: 1M) to limit the number of caps a client session may hold. Long running client sessions with a large number of caps have been a source of instability in the MDS when all of these caps need to be processed during certain session events. It is recommended to not unnecessarily increase this value. - The “mds_recall_state_timeout” config parameter has been removed. Late client recall warnings are now generated based on the number of caps the MDS has recalled which have not been released. The new config parameters “mds_recall_warning_threshold” (default: 32K) and “mds_recall_warning_decay_rate” (default: 60s) set the threshold for this warning. - The “cache drop” admin socket command has been removed. The “ceph tell mds.X cache drop” remains. OSD: - A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option “mon_warn_on_slow_ping_ratio” specifies a percentage of “osd_heartbeat_grace” to determine the threshold. A value of zero disables the warning. A new configuration option “mon_warn_on_slow_ping_time”, specified in milliseconds, overrides the computed value, causing a warning when OSD heartbeat pings take longer than the specified amount. A new admin command “ceph daemon mgr.# dump_osd_network [threshold]” lists all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. A new admin command ceph daemon osd.# dump_osd_network [threshold]” does the same but only including heartbeats initiated by the specified OSD. - The default value of the “osd_deep_scrub_large_omap_object_key_threshold” parameter has been lowered to detect an object with large number of omap keys more easily. RGW: - radosgw-admin introduces two subcommands that allow the managing of expire-stale objects that might be left behind after a bucket reshard in earlier versions of RGW. One subcommand lists such objects and the other deletes them. Read the troubleshooting section of the dynamic resharding docs for details.

4 7

Ceph User Survey 2019
by Mike Perez 12 Dec '19

12 Dec '19

Hi Cephers, To better understand how our current users utilize Ceph, we conducted a public community survey. This information is a guide to the community of how we spend our contribution efforts for future development. The survey results will remain anonymous and aggregated in future Ceph Foundation publications to the community. I'm pleased to announce after much discussion on the Ceph dev mailing list [0] that the community has formed the Ceph Survey for 2019. The deadline for this survey due to it being out later than we'd like will be January 31st, 2020 at 11:59 PT. https://ceph.io/user-survey/ We have discussed in the future to use the Ceph telemetry module to collect the data to save time for our users. Please let me know of any mistakes that need to be corrected on the survey. Thanks! [0] - https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/WU374ZJP5N3NKY22X2… -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

4 3

osdmaps not trimmed until ceph-mon's restarted (if cluster has a down osd)
by Dan van der Ster 09 Dec '19

09 Dec '19

Hi Joao, I might have found the reason why several of our clusters (and maybe Bryan's too) are getting stuck not trimming osdmaps. It seems that when an osd fails, the min_last_epoch_clean gets stuck forever (even long after HEALTH_OK), until the ceph-mons are restarted. I've updated the ticket: https://tracker.ceph.com/issues/41154 Cheers, Dan

4 4

Behavior of EC pool when a host goes offline
by majia xiao 03 Dec '19

03 Dec '19

Hi all, We have a Ceph（version 12.2.4）cluster that adopts EC pools, and it consists of 10 hosts for OSDs. The corresponding commands to create the EC pool are listed as follows: ceph osd erasure-code-profile set profile_jerasure_4_3_reed_sol_van \ plugin=jerasure \ k=4 \ m=3 \ technique=reed_sol_van \ packetsize=2048 \ crush-device-class=hdd \ crush-failure-domain=host ceph osd pool create pool_jerasure_4_3_reed_sol_van 2048 2048 erasure profile_jerasure_4_3_reed_sol_van Since that the EC pool's crush-failure-domain is configured to be "host", we just disable the network interfaces of some hosts (using "ifdown" command) to verify the functionality of the EC pool. And here are the phenomena we have observed: First of all, the IO rate (of "rados bench", which we used for benchmark) drops immediately to 0 when one host goes offline. Secondly, it takes a lot of time (around 100 seconds) for Ceph to detect corresponding OSDs on that host are down. Finally, once the Ceph has detected all offline OSDs, the EC pool seems to act normally and it is ready for IO operations again. So, here are my questions: 1. Is this normal that the IO rate drops to 0 immediately even though there is only one host goes offline? 2. How to make Ceph reduce the time needed to detect failed OSDs? Thanks for any help. Best regards, Majia Xiao

2 1

Possible data corruption with 14.2.3 and 14.2.4
by Sage Weil 03 Dec '19

03 Dec '19

Hi everyone, We've identified a data corruption bug[1], first introduced[2] (by yours truly) in 14.2.3 and affecting both 14.2.3 and 14.2.4. The corruption appears as a rocksdb checksum error or assertion that looks like os/bluestore/fastbmap_allocator_impl.h: 750: FAILED ceph_assert(available >= allocated) or in some cases a rocksdb checksum error. It only affects BlueStore OSDs that have a separate 'db' or 'wal' device. We have a fix[3] that is working its way through testing, and will expedite the next Nautilus point release (14.2.5) once it is ready. If you are running 14.2.2 or 14.2.1 and use BlueStore OSDs with separate 'db' volumes, you should consider waiting to upgrade until 14.2.5 is released. A big thank you to Igor Fedotov and several *extremely* helpful users who managed to reproduce and track down this problem! sage [1] https://tracker.ceph.com/issues/42223 [2] https://github.com/ceph/ceph/commit/096033b9d931312c0688c2eea7e14626bfde0ad… [3] https://github.com/ceph/ceph/pull/31621

5 8

Dual network board setup info
by Rodrigo Severo - Fábrica 02 Dec '19

02 Dec '19

Hi, I have a CephFS instance and I am also planning on also deploying an Object Storage interface. My servers have 2 network boards each. I would like to use the current local one to talk to Cephs clients (both CephFS and Object Storage) and use the second one to all Cephs processes to talk one to the other. I'm quite sure that Ceph has support for this kind of setup but I can't find how to do such a thing. I already have a "public network" setting that, AFAIU, sets the interface used by Ceph to talk to clients but I don't know if it defines CephFS MDS, RGW, or all of them. And I can't see how to make the different Ceph processes to talk to each other through the other interface. I found "mon_host" setting but should I change it to get what I want? And what else? Is there some example of such a setup that I could learn from? Or maybe someone would be kind enough to scratch a basic config to achieve this setup? Or maybe there is some documentation that deals with such a scenario that I haven't found. Well, I'm in search for more info, any kind. Thanks in advance for your help and attention, Rodrigo Severo

4 7

Not able to create and remove snapshots
by Marc Roos 30 Nov '19

30 Nov '19

I am not sure since when, but I am not able to create nor delete snapshots anymore. I am getting a permission denied. I upgraded recently from Luminous to Nautilus and set this allow_new_snaps as mentioned on[1] [@ .snap]# ls snap-1 snap-2 snap-3 snap-4 snap-5 snap-6 snap-7 [@ .snap]# rmdir snap-7 rmdir: failed to remove ‘snap-7’: Permission denied [@ .snap]# mkdir snap-8 mkdir: cannot create directory ‘snap-8’: Permission denied [1] https://docs.ceph.com/docs/nautilus/dev/cephfs-snapshots/

1 1

ceph auth
by Marc Roos 30 Nov '19

30 Nov '19

Is there a ceph auth command that just list all clients? Without dumping keys to the console. I would recommend making 'ceph auth ls' just display client names/ids. If you want the key, there are already other commands.

1 2

Questions about the EC pool
by majia xiao 29 Nov '19

29 Nov '19

Hello, We have a Ceph cluster (version 12.2.4) with 10 hosts, and there are 21 OSDs on each host. An EC pool is created with the following commands: ceph osd erasure-code-profile set profile_jerasure_4_3_reed_sol_van \ plugin=jerasure \ k=4 \ m=3 \ technique=reed_sol_van \ packetsize=2048 \ crush-device-class=hdd \ crush-failure-domain=host ceph osd pool create pool_jerasure_4_3_reed_sol_van 2048 2048 erasure profile_jerasure_4_3_reed_sol_van Here are my questions: 1. The EC pool is created using k=4, m=3, and crush-device-class=hdd, so we just disable the network interfaces of some hosts (using "ifdown" command) to verify the functionality of the EC pool while performing ‘rados bench’ command. However, the IO rate drops immediately to 0 when a single host goes offline, and it takes a long time (~100 seconds) for the IO rate becoming normal. As far as I know, the default value of min_size is k+1 or 5, which means that the EC pool can be still working even if there are two hosts offline. Is there something wrong with my understanding? 2. According to our observations, it seems that the IO rate becomes normal when Ceph detects all OSDs corresponding to the failed host. Is there any way to reduce the time needed for Ceph to detect all failed OSDs? Thanks for any help. Best regards, Majia Xiao

2 1

Re: mimic 13.2.6 too much broken connexions
by Vincent Godin 29 Nov '19

29 Nov '19

If it was a network issue, the counters should explose (as i said, with a log level of 5 on the messenger, we observed more then 80 000 lossy channels per minute) but nothing abnormal is relevant on the counters (on switchs and servers) On the switchs no drop, no crc error, no packet loss, only some output discards but not enough to be significant. On the NICs on the servers via ethtool -S, nothing is relevant. And as i said, an other mimic cluster with different hardware has the same behavior Ceph uses connexions pools from host to host but how does it check the availability of these connexions over the time ? And as the network doesn't seem to be guilty, what can explain these broken channels ? Le mer. 27 nov. 2019 à 19:05, Anthony D'Atri <aad(a)dreamsnake.net> a écrit : > > Are you bonding NIC ports? If so do you have the correct hash policy defined? Have you looked at the *switch* side for packet loss, CRC errors, etc? What you report could be consistent with this. Since the host interface for a given connection will vary by the bond hash, some OSD connections will use one port and some the other. So if one port has switch side errors, or is blackholed on the switch, you could see some heart beating impacted but not others. > > Also make sure you have the optimal reporters value. > > > On Nov 27, 2019, at 7:31 AM, Vincent Godin <vince.mlist(a)gmail.com> wrote: > > > > Till i submit the mail below few days ago, we found some clues > > We observed a lot of lossy connexion like : > > ceph-osd.9.log:2019-11-27 11:03:49.369 7f6bb77d0700 0 -- > > 192.168.4.181:6818/2281415 >> 192.168.4.41:0/1962809518 > > conn(0x563979a9f600 :6818 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH > > pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) > > channel (new one lossy=1) > > We raised the log of the messenger to 5/5 and observed for the whole > > cluster more than 80 000 lossy connexion per minute !!! > > We adjusted the "ms_tcp_read_timeout" from 900 to 60 sec then no more > > lossy connexion in logs nor health check failed > > It's just a workaround but there is a real problem with these broken > > sessions and it leads to two > > assertions : > > - Ceph take too much time to detect broken session and should recycle quicker ! > > - The reasons for these broken sessions ? > > > > We have a other mimic cluster on different hardware and observed the > > same behavior : lot of lossy sessions, slow ops and co. > > Symptoms are the same : > > - some OSDs on one host have no response from an other osd on a different hosts > > - after some time, slow ops are detected > > - sometime it leads to ioblocked > > - after about 15mn the problem vanish > > > > ----------- > > > > Help on diag needed : heartbeat_failed > > > > We encounter a strange behavior on our Mimic 13.2.6 cluster. A any > > time, and without any load, some OSDs become unreachable from only > > some hosts. It last 10 mn and then the problem vanish. > > It 's not always the same OSDs and the same hosts. There is no network > > failure on any of the host (because only some OSDs become unreachable) > > nor disk freeze as we can see in our grafana dashboard. Logs message > > are : > > first msg : > > 2019-11-24 09:19:43.292 7fa9980fc700 -1 osd.596 146481 > > heartbeat_check: no reply from 192.168.6.112:6817 osd.394 since back > > 2019-11-24 09:19:22.761142 front 2019-11-24 09:19:39.769138 (cutoff > > 2019-11-24 09:19:23.293436) > > last msg: > > 2019-11-24 09:30:33.735 7f632354f700 -1 osd.591 146481 > > heartbeat_check: no reply from 192.168.6.123:6828 osd.600 since back > > 2019-11-24 09:27:05.269330 front 2019-11-24 09:30:33.214874 (cutoff > > 2019-11-24 09:30:13.736517) > > During this time, 3 hosts were involved : host-18, host-20 and host-30 : > > host-30 is the only one who can't see osds 346,356,and 352 on host-18 > > host-30 is the only one who can't see osds 387 and 394 on host-20 > > host-18 is the only one who can't see osds 583, 585, 591 and 597 on host-30 > > We can't see any strange behavior on hosts 18, 20 and 30 in our node > > exporter data during this time > > Any ideas or advices ?

2 2

How to set size for CephFs
by Alokkumar Mahajan 28 Nov '19

28 Nov '19

Hello, I am new to Ceph and currently i am working on setting up CephFs and RBD environment. I have successfully setup Ceph Cluster with 4 OSD's (2 OSD's with size 50GB and 2 OSD's with size 300GB). But while setting up CephFs the size which i see allocated for CephFs Data and metadata pools is 55GB. But i want to have 300GB assigned for CephFs. I tried using "target_size_bytes" flag while creating pool but it is not working (it saus invalid command). Same result when i use target_size_bytes with (ceph osd pool set) after creating pool. I am not sure if i am doing something silly here. Can someone please guide me on this? Thanks in adv.!

3 5

Re: cephfs worm feature
by j j 28 Nov '19

28 Nov '19

Thanks for the information, I'll take a look at this pr and think it over. Jeff Layton <jlayton(a)redhat.com> 于2019年11月27日周三下午6:50写道： > On Wed, 2019-11-27 at 15:14 +0800, j j wrote: > > Hi all, > > > > Recently I encountered a situation requires reliable file storage > with cephfs, and the point is those data is not allowed to get modified or > deleted. > > After some learning I found that the WORM(write once read many) feature > is exactly what I need.Unfortunately, as far as I know, there is no worm > > feature in cephfs. > > So I was wondering is there any plan or design about this feature? > > > > Thanks. > > There's a pull request for this that has been stalled since spring: > > https://github.com/ceph/ceph/pull/26691 > > Personally, I don't see how we can get away with making file data 100% > immutable. We'll need to allow _some_ entity to un-WORM the thing, and > it was never clear to me how that would work. > -- > Jeff Layton <jlayton(a)redhat.com> > >

1 0

mimic 13.2.6 too much broken connexions
by Vincent Godin 27 Nov '19

27 Nov '19

Till i submit the mail below few days ago, we found some clues We observed a lot of lossy connexion like : ceph-osd.9.log:2019-11-27 11:03:49.369 7f6bb77d0700 0 -- 192.168.4.181:6818/2281415 >> 192.168.4.41:0/1962809518 conn(0x563979a9f600 :6818 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) We raised the log of the messenger to 5/5 and observed for the whole cluster more than 80 000 lossy connexion per minute !!! We adjusted the "ms_tcp_read_timeout" from 900 to 60 sec then no more lossy connexion in logs nor health check failed It's just a workaround but there is a real problem with these broken sessions and it leads to two assertions : - Ceph take too much time to detect broken session and should recycle quicker ! - The reasons for these broken sessions ? We have a other mimic cluster on different hardware and observed the same behavior : lot of lossy sessions, slow ops and co. Symptoms are the same : - some OSDs on one host have no response from an other osd on a different hosts - after some time, slow ops are detected - sometime it leads to ioblocked - after about 15mn the problem vanish ----------- Help on diag needed : heartbeat_failed We encounter a strange behavior on our Mimic 13.2.6 cluster. A any time, and without any load, some OSDs become unreachable from only some hosts. It last 10 mn and then the problem vanish. It 's not always the same OSDs and the same hosts. There is no network failure on any of the host (because only some OSDs become unreachable) nor disk freeze as we can see in our grafana dashboard. Logs message are : first msg : 2019-11-24 09:19:43.292 7fa9980fc700 -1 osd.596 146481 heartbeat_check: no reply from 192.168.6.112:6817 osd.394 since back 2019-11-24 09:19:22.761142 front 2019-11-24 09:19:39.769138 (cutoff 2019-11-24 09:19:23.293436) last msg: 2019-11-24 09:30:33.735 7f632354f700 -1 osd.591 146481 heartbeat_check: no reply from 192.168.6.123:6828 osd.600 since back 2019-11-24 09:27:05.269330 front 2019-11-24 09:30:33.214874 (cutoff 2019-11-24 09:30:13.736517) During this time, 3 hosts were involved : host-18, host-20 and host-30 : host-30 is the only one who can't see osds 346,356,and 352 on host-18 host-30 is the only one who can't see osds 387 and 394 on host-20 host-18 is the only one who can't see osds 583, 585, 591 and 597 on host-30 We can't see any strange behavior on hosts 18, 20 and 30 in our node exporter data during this time Any ideas or advices ?

1 0

EC PGs stuck activating, 2^31-1 as OSD ID, automatic recovery not kicking in
by Zoltan Arnold Nagy 27 Nov '19

27 Nov '19

Hi, We have a cluster where we mix HDDs and NVMe drives using device classes with a specific crush role for each class. One of our NVMe drives physically died which caused some of our PGs to go into this state: pg 26.ac is stuck undersized for 60830.991784, current state activating+undersized+degraded+remapped, last acting [353,373,368,377,2147483647,350] pg 26.d1 is stuck undersized for 60830.587711, current state activating+undersized+degraded+remapped, last acting [343,2147483647,347,358,366,355] pg 26.e1 is stuck undersized for 60830.980585, current state activating+undersized+degraded+remapped, last acting [340,349,370,2147483647,360,376] ... and so on. Recovery never happened and we had to manually restart all affected OSDs for all PGs stuck in such a state. The 2^31-1 in there seems to indicate an overflow somewhere - the way we were able to figure out where exactly is to query the PG and compare the "up" and "acting" sets - only _one_ of them had the 2^31-1 number in place of the correct OSD number. We restarted that and the PG started doing its job and recovered. The issue seems to be going back to 2015: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-May/001661.html however no solution... I'm more concerned about the cluster not being able to recover (it's a 4+2 EC pool across 12 hosts - plenty of room to heal) than about the weird print-out. The VMs who wanted to access data in any of the affected PGs of course died. Are we missing some settings to let the cluster self-heal even for EC pools? First EC pool in production :) Cheers, Zoltan

4 4

EC pool used space high
by Erdem Agaoglu 27 Nov '19

27 Nov '19

Hi, I have a rook-provisioned cluster to be used for RBDs only. I have 2 pools named replicated-metadata-pool and ec-data-pool. EC parameters are 6+3. I've been writing some data to this cluster for some time and noticed that the reported usage is not what I was expecting. # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.4 PiB 4.3 PiB 1.2 PiB 1.2 PiB 21.77 TOTAL 5.4 PiB 4.3 PiB 1.2 PiB 1.2 PiB 21.77 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL replicated-metadata-pool 1 90 KiB 408 38 MiB 0 1.2 PiB ec-data-pool 2 722 TiB 191.64M 1.2 PiB 25.04 2.4 PiB Since these numbers are rounded a bit too much, I generally use prometheus metrics on mgr, which are as follows: ceph_pool_stored : 793,746 G for ec-data-pool and 92323 for replicated-metadata-pool ceph_pool_stored_raw: 1,190,865 G for ec-data-pool and 99213 for replicated-metadata-pool ceph_cluster_total_used_bytes: 1,329,374 G ceph_cluster_total_used_raw_bytes: 1,333,013 G sum(ceph_bluefs_db_used_bytes) : 3,638 G So ceph_pool_stored for the EC pool is a bit higher than the total used space of the formatted RBDs. I think that's because of the sparse nature and deleted blocks not being fstrimmed yet. That's OK. ceph_pool_stored_raw is almost exactly 1.5x ceph_pool_stored which is what I'd expect considering EC parameters of 6+3. What I can't find is the 138,509 G difference between the ceph_cluster_total_used_bytes and ceph_pool_stored_raw. This is not static BTW, checking the same data historically shows we have about 1.12x of what we expect. This seems to make our 1.5x EC overhead a 1.68x overhead in reality. Anyone have any ideas for why this is the case? We also have a ceph_cluster_total_used_raw_bytes metric, I believe to be close to data+metadata. Which is why I tried to show with sum(ceph_bluefs_db_used_bytes). Is that correct? Best, -- erdem agaoglu

3 5

[radosgw-admin] Unable to Unlink Bucket From UID
by Mac Wynkoop 26 Nov '19

26 Nov '19

Hi all, I seem to be running into an issue when attempting to unlink a bucket from a user; this is my output: user@server ~ $ radosgw-admin bucket unlink --bucket=user_5493/LF-Store --uid=user_5493 failure: 2019-11-26 15:19:48.689 7fda1c2009c0 0 bucket entry point user mismatch, can't unlink bucket: user_5493$BRTC != user_5493 (22) Invalid argument user@server ~ $ I did some searching around, and no one seems to have seen this before. Any ideas? Thanks, Mac

1 0

rbd lvm xfs fstrim vs rbd xfs fstrim
by Marc Roos 26 Nov '19

26 Nov '19

If I do an fstrim /mount/fs and this is an xsf directly on a rbd device. I can see space being freed instantly with eg. rbd du. However when there is an lvm in between, it looks like this is not freed. I already enabled issue_discards = 1 in lvm.conf but as the comment says, probably only in case of lvremove. Is it possible to get fstrim working with lvm?

1 1

ceph user list respone
by Frank R 26 Nov '19

26 Nov '19

Hi, For your response: "You should use not more 1Gb for WAL and 30Gb for RocksDB. Numbers ! 3,30,300 (Gb) for block.db is useless. " Do you mean the block.db size should be 3, 30 or 300GB and nothing else? If so, thy not? Thanks, Frank

2 1

ceph-fuse non-privileged user mount
by yi zhang 26 Nov '19

26 Nov '19

Hi, Recently, I'm trying to mount cephfs using non-privileged users via ceph-fuse，but it‘s always fail. I looked at the code and found that there will be a remount operation when using ceph-fuse mount. Remount will execute the 'mount -i -o remount {mountpoint}' command and that causes the mount to fail. I want to ask How to use non-privileged users to mount cephfs via ceph-fuse? Thanks.

1 0

Single mount X multiple mounts
by Rodrigo Severo - Fábrica 26 Nov '19

26 Nov '19

Hi, Just starting to use CephFS. I would like to know the impacts of having one single CephFS mount X having several. If I have several subdirectories in my CephFS that should be accessible to different users, with users needing access to different sets of mounts, would it be important for me to try to predefine these sets so to minimize the number of mounts each user needs or can I treat each subdirectory as an independent entity and just mount all necessary for each user and, so, increasing the number of different mounts for each user? Regards, Rodrigo Severo

2 1

ceph cache pool question
by Shawn A Kwang 25 Nov '19

25 Nov '19

I have a question about ceph cache pools as documented on this page: https://docs.ceph.com/docs/nautilus/dev/cache-pool/ Is the cache pool feature still considered a good idea? Reading some of the email archives I find some discussion of how this caching is not recommended anymore, for version=nautilus. Is that correct? What is my use-case? We are using cephfs, and have a large cephfs pool of HDD with some NVMe drives for bluestore WAL and DB. Total storage is 2.1 PB. Replication=3. There is also a separate metadata pool, as required for cephfs. We have a spare storage server with about 14 TB of NVME drives. Would it be worthwhile to setup an NVMe cache pool for the main cephfs pool? Sincerely, Shawn Kwang -- Associate Scientist Center for Gravitation, Cosmology, and Astrophysics University of Wisconsin-Milwaukee office: +1 414 229 4960 kwangs(a)uwm.edu

1 0

FUSE X kernel mounts
by Rodrigo Severo - Fábrica 25 Nov '19

25 Nov '19

Hi, I'm just deploying a CephFS service. I would like to know the expected differences between a FUSE and a kernel mount. Why the 2 options? When should I use one and when should I use the other? Regards, Rodrigo Severo

3 3

Ceph manager not starting
by Romain Raynaud 24 Nov '19

24 Nov '19

Hello please can you help me with that. I'm on ubuntu 18.04 bionic trying to install ceph version 12.2.12 with ceph-deploy to 3 nodes on odroid xu4 (armhf). Creating osd with lvm volumes seem's to work but after starting manager by the command "ceph-deploy mgr create node1", mgr go to starting mode then stay in a health warn state due to no mgr demons active. regards --$ ceph -s --------------------------------------------------------------------- cluster: id: 84f324c2-ac27-4f72-bcb0-7ff1355ee97e health: HEALTH_WARN no active mgr services: mon: 3 daemons, quorum node1,node3,node2 mgr: no daemons active osd: 3 osds: 3 up, 3 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0B usage: 0B used, 0B / 0B avail pgs: ---/var/log/ceph/ceph-mgr.node1.log------------------------------------------------------------- 2019-11-14 11:08:45.794236 b6f87230 0 ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable), process ceph-mgr, pid 7879 2019-11-14 11:08:45.799472 b6f87230 0 pidfile_write: ignore empty --pid-file 2019-11-14 11:08:45.839575 b6f87230 1 mgr send_beacon standby 2019-11-14 11:08:45.866590 b06c9c30 -1 *** Caught signal (Segmentation fault) ** in thread b06c9c30 thread_name:ms_dispatch ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x302eac) [0x7d6eac] 2: (()+0x25750) [0xb6862750] 3: (_ULarm_step()+0x5b) [0xb67eecec] 4: (()+0x255e8) [0xb6cb05e8] 5: (GetStackTrace(void**, int, int)+0x25) [0xb6cb0a3e] 6: (tcmalloc::PageHeap::GrowHeap(unsigned int)+0xb9) [0xb6ca536a] 7: (tcmalloc::PageHeap::New(unsigned int)+0x79) [0xb6ca55e6] 8: (tcmalloc::CentralFreeList::Populate()+0x71) [0xb6ca45ce] 9: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x1b) [0xb6ca4760] 10: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x6d) [0xb6ca47de] 11: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, unsigned int)+0x51) [0xb6ca6a56] 12: (malloc()+0x22d) [0xb6cb1a8e] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -108> 2019-11-14 11:08:45.776013 b6f87230 5 asok(0x5625320) register_command perfcounters_dump hook 0x55bc090 -107> 2019-11-14 11:08:45.776070 b6f87230 5 asok(0x5625320) register_command 1 hook 0x55bc090 -106> 2019-11-14 11:08:45.776096 b6f87230 5 asok(0x5625320) register_command perf dump hook 0x55bc090 -105> 2019-11-14 11:08:45.776134 b6f87230 5 asok(0x5625320) register_command perfcounters_schema hook 0x55bc090 -104> 2019-11-14 11:08:45.776155 b6f87230 5 asok(0x5625320) register_command perf histogram dump hook 0x55bc090 -103> 2019-11-14 11:08:45.776194 b6f87230 5 asok(0x5625320) register_command 2 hook 0x55bc090 -102> 2019-11-14 11:08:45.776244 b6f87230 5 asok(0x5625320) register_command perf schema hook 0x55bc090 -101> 2019-11-14 11:08:45.776267 b6f87230 5 asok(0x5625320) register_command perf histogram schema hook 0x55bc090 -100> 2019-11-14 11:08:45.776304 b6f87230 5 asok(0x5625320) register_command perf reset hook 0x55bc090 -99> 2019-11-14 11:08:45.776325 b6f87230 5 asok(0x5625320) register_command config show hook 0x55bc090 -98> 2019-11-14 11:08:45.776358 b6f87230 5 asok(0x5625320) register_command config help hook 0x55bc090 -97> 2019-11-14 11:08:45.776380 b6f87230 5 asok(0x5625320) register_command config set hook 0x55bc090 -96> 2019-11-14 11:08:45.776416 b6f87230 5 asok(0x5625320) register_command config get hook 0x55bc090 -95> 2019-11-14 11:08:45.776457 b6f87230 5 asok(0x5625320) register_command config diff hook 0x55bc090 -94> 2019-11-14 11:08:45.776479 b6f87230 5 asok(0x5625320) register_command config diff get hook 0x55bc090 -93> 2019-11-14 11:08:45.776513 b6f87230 5 asok(0x5625320) register_command log flush hook 0x55bc090 -92> 2019-11-14 11:08:45.776534 b6f87230 5 asok(0x5625320) register_command log dump hook 0x55bc090 -91> 2019-11-14 11:08:45.776568 b6f87230 5 asok(0x5625320) register_command log reopen hook 0x55bc090 -90> 2019-11-14 11:08:45.776679 b6f87230 5 asok(0x5625320) register_command dump_mempools hook 0x5757b04 -89> 2019-11-14 11:08:45.794236 b6f87230 0 ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable), process ceph-mgr, pid 7879 -88> 2019-11-14 11:08:45.799472 b6f87230 0 pidfile_write: ignore empty --pid-file -87> 2019-11-14 11:08:45.800376 b6f87230 1 finished global_init_daemonize -86> 2019-11-14 11:08:45.813287 b6f87230 5 asok(0x5625320) init /var/run/ceph/ceph-mgr.node1.asok -85> 2019-11-14 11:08:45.813359 b6f87230 5 asok(0x5625320) bind_and_listen /var/run/ceph/ceph-mgr.node1.asok -84> 2019-11-14 11:08:45.813867 b6f87230 5 asok(0x5625320) register_command 0 hook 0x55bc180 -83> 2019-11-14 11:08:45.813920 b6f87230 5 asok(0x5625320) register_command version hook 0x55bc180 -82> 2019-11-14 11:08:45.813954 b6f87230 5 asok(0x5625320) register_command git_version hook 0x55bc180 -81> 2019-11-14 11:08:45.813994 b6f87230 5 asok(0x5625320) register_command help hook 0x55bc178 -80> 2019-11-14 11:08:45.814085 b6f87230 5 asok(0x5625320) register_command get_command_descriptions hook 0x55bc170 -79> 2019-11-14 11:08:45.814371 b3ed0c30 5 asok(0x5625320) entry start -78> 2019-11-14 11:08:45.819066 b36cfc30 2 Event(0x55be068 nevent=5000 time_id=1).set_owner idx=0 owner=3010264112 -77> 2019-11-14 11:08:45.819281 b2ecec30 2 Event(0x55be488 nevent=5000 time_id=1).set_owner idx=1 owner=3001871408 -76> 2019-11-14 11:08:45.819484 b26cdc30 2 Event(0x55be1c8 nevent=5000 time_id=1).set_owner idx=2 owner=2993478704 -75> 2019-11-14 11:08:45.821210 b6f87230 1 Processor -- start -74> 2019-11-14 11:08:45.821472 b6f87230 1 -- - start start -73> 2019-11-14 11:08:45.821506 b6f87230 10 monclient: build_initial_monmap -72> 2019-11-14 11:08:45.821667 b6f87230 10 monclient: init -71> 2019-11-14 11:08:45.821796 b6f87230 5 adding auth protocol: cephx -70> 2019-11-14 11:08:45.821826 b6f87230 10 monclient: auth_supported 2 method cephx -69> 2019-11-14 11:08:45.822606 b6f87230 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-node1/keyring -68> 2019-11-14 11:08:45.822947 b6f87230 10 monclient: _reopen_session rank -1 -67> 2019-11-14 11:08:45.823131 b6f87230 10 monclient(hunting): picked mon.noname-b con 0x5804d00 addr [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 -66> 2019-11-14 11:08:45.823297 b6f87230 10 monclient(hunting): picked mon.noname-c con 0x5805a00 addr [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -65> 2019-11-14 11:08:45.823437 b6f87230 1 -- - --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5600680 con 0 -64> 2019-11-14 11:08:45.823555 b6f87230 1 -- - --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5600820 con 0 -63> 2019-11-14 11:08:45.823646 b6f87230 10 monclient(hunting): _renew_subs -62> 2019-11-14 11:08:45.825773 b26cdc30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 learned_addr learned my addr [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 -61> 2019-11-14 11:08:45.826773 b26cdc30 2 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -60> 2019-11-14 11:08:45.827140 b2ecec30 2 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -59> 2019-11-14 11:08:45.828839 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 1 0x5630900 mon_map magic: 0 v1 -58> 2019-11-14 11:08:45.829033 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.1 [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 1 ==== mon_map magic: 0 v1 ==== 469+0+0 (726272991 0 0) 0x5630900 con 0x5805a00 -57> 2019-11-14 11:08:45.829141 b06c9c30 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -56> 2019-11-14 11:08:45.829085 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 2 0x560e540 auth_reply(proto 2 0 (0) Success) v1 -55> 2019-11-14 11:08:45.829165 b2ecec30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1). rx mon.2 seq 1 0x5630a80 mon_map magic: 0 v1 -54> 2019-11-14 11:08:45.829225 b06c9c30 10 monclient(hunting): got monmap 1, mon.noname-c is now rank -1 -53> 2019-11-14 11:08:45.829244 b06c9c30 10 monclient(hunting): dump: epoch 1 fsid 84f324c2-ac27-4f72-bcb0-7ff1355ee97e last_changed 2019-11-13 14:18:49.133327 created 2019-11-13 14:18:49.133327 0: [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:6789/0 mon.node1 1: [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 mon.fpgh3 2: [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 mon.fpgh2 -52> 2019-11-14 11:08:45.829331 b2ecec30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1). rx mon.2 seq 2 0x560e700 auth_reply(proto 2 0 (0) Success) v1 -51> 2019-11-14 11:08:45.829402 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.1 [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3710817385 0 0) 0x560e540 con 0x5805a00 -50> 2019-11-14 11:08:45.829512 b06c9c30 10 monclient(hunting): my global_id is 24280 -49> 2019-11-14 11:08:45.829791 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x5600d00 con 0 -48> 2019-11-14 11:08:45.829910 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.2 [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 1 ==== mon_map magic: 0 v1 ==== 469+0+0 (726272991 0 0) 0x5630a80 con 0x5804d00 -47> 2019-11-14 11:08:45.829982 b06c9c30 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -46> 2019-11-14 11:08:45.830052 b06c9c30 10 monclient(hunting): got monmap 1, mon.fpgh2 is now rank 2 -45> 2019-11-14 11:08:45.830085 b06c9c30 10 monclient(hunting): dump: epoch 1 fsid 84f324c2-ac27-4f72-bcb0-7ff1355ee97e last_changed 2019-11-13 14:18:49.133327 created 2019-11-13 14:18:49.133327 0: [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:6789/0 mon.node1 1: [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 mon.fpgh3 2: [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 mon.fpgh2 -44> 2019-11-14 11:08:45.830244 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.2 [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2346605455 0 0) 0x560e700 con 0x5804d00 -43> 2019-11-14 11:08:45.830343 b06c9c30 10 monclient(hunting): my global_id is 24308 -42> 2019-11-14 11:08:45.830535 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x5600680 con 0 -41> 2019-11-14 11:08:45.832185 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 3 0x560e8c0 auth_reply(proto 2 0 (0) Success) v1 -40> 2019-11-14 11:08:45.832327 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.1 [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (3339254479 0 0) 0x560e8c0 con 0x5805a00 -39> 2019-11-14 11:08:45.832834 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x5600d00 con 0 -38> 2019-11-14 11:08:45.833078 b2ecec30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1). rx mon.2 seq 3 0x560e700 auth_reply(proto 2 0 (0) Success) v1 -37> 2019-11-14 11:08:45.833228 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.2 [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (3081092587 0 0) 0x560e700 con 0x5804d00 -36> 2019-11-14 11:08:45.833816 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x5600820 con 0 -35> 2019-11-14 11:08:45.836366 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 4 0x560ea80 auth_reply(proto 2 0 (0) Success) v1 -34> 2019-11-14 11:08:45.836509 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.1 [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 751+0+0 (1301292739 0 0) 0x560ea80 con 0x5805a00 -33> 2019-11-14 11:08:45.837249 b2ecec30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1). rx mon.2 seq 4 0x560e8c0 auth_reply(proto 2 0 (0) Success) v1 -32> 2019-11-14 11:08:45.837359 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_OPEN pgs=87 cs=1 l=1).mark_down -31> 2019-11-14 11:08:45.837425 b06c9c30 2 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_OPEN pgs=87 cs=1 l=1)._stop -30> 2019-11-14 11:08:45.837558 b06c9c30 1 monclient: found mon.fpgh3 -29> 2019-11-14 11:08:45.837623 b06c9c30 10 monclient: _send_mon_message to mon.fpgh3 at [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -28> 2019-11-14 11:08:45.837669 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- mon_subscribe({mgrmap=0+,monmap=0+}) v2 -- 0x55beb00 con 0 -27> 2019-11-14 11:08:45.837797 b06c9c30 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-11-14 11:08:15.837792) -26> 2019-11-14 11:08:45.837860 b06c9c30 10 monclient: _send_mon_message to mon.fpgh3 at [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -25> 2019-11-14 11:08:45.837923 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x5600680 con 0 -24> 2019-11-14 11:08:45.838036 b6f87230 5 monclient: authenticate success, global_id 24280 -23> 2019-11-14 11:08:45.838182 b6f87230 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -22> 2019-11-14 11:08:45.838290 b6f87230 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -21> 2019-11-14 11:08:45.838619 b6f87230 5 asok(0x5625320) register_command objecter_requests hook 0x55bc1d8 -20> 2019-11-14 11:08:45.838736 b6f87230 10 monclient: _renew_subs -19> 2019-11-14 11:08:45.838770 b6f87230 10 monclient: _send_mon_message to mon.fpgh3 at [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -18> 2019-11-14 11:08:45.838816 b6f87230 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- mon_subscribe({osdmap=0}) v2 -- 0x55bec60 con 0 -17> 2019-11-14 11:08:45.839247 b6f87230 5 asok(0x5625320) register_command mds_requests hook 0xbe83e770 -16> 2019-11-14 11:08:45.839310 b6f87230 5 asok(0x5625320) register_command mds_sessions hook 0xbe83e770 -15> 2019-11-14 11:08:45.839347 b6f87230 5 asok(0x5625320) register_command dump_cache hook 0xbe83e770 -14> 2019-11-14 11:08:45.839383 b6f87230 5 asok(0x5625320) register_command kick_stale_sessions hook 0xbe83e770 -13> 2019-11-14 11:08:45.839420 b6f87230 5 asok(0x5625320) register_command status hook 0xbe83e770 -12> 2019-11-14 11:08:45.839575 b6f87230 1 mgr send_beacon standby -11> 2019-11-14 11:08:45.839646 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 5 0x55fde00 mgrmap(e 11813) v1 -10> 2019-11-14 11:08:45.839756 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.1 [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 5 ==== mgrmap(e 11813) v1 ==== 237+0+0 (2937343514 0 0) 0x55fde00 con 0x5805a00 -9> 2019-11-14 11:08:45.839778 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 6 0x5630900 mon_map magic: 0 v1 -8> 2019-11-14 11:08:45.840302 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 7 0x560e700 auth_reply(proto 2 0 (0) Success) v1 -7> 2019-11-14 11:08:45.840645 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 8 0x5600680 osd_map(45..45 src has 1..45) v3 -6> 2019-11-14 11:08:45.840927 b6f87230 10 monclient: _send_mon_message to mon.fpgh3 at [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -5> 2019-11-14 11:08:45.840980 b6f87230 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- mgrbeacon mgr.node1(84f324c2-ac27-4f72-bcb0-7ff1355ee97e,24280, -, 0) v6 -- 0x55d6400 con 0 -4> 2019-11-14 11:08:45.841111 b6f87230 4 mgr init Complete. -3> 2019-11-14 11:08:45.841220 b06c9c30 4 mgr ms_dispatch standby mgrmap(e 11813) v1 -2> 2019-11-14 11:08:45.841266 b06c9c30 4 mgr handle_mgr_map received map epoch 11813 -1> 2019-11-14 11:08:45.841282 b06c9c30 4 mgr handle_mgr_map active in map: 0 active is 0 0> 2019-11-14 11:08:45.866590 b06c9c30 -1 *** Caught signal (Segmentation fault) ** in thread b06c9c30 thread_name:ms_dispatch ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x302eac) [0x7d6eac] 2: (()+0x25750) [0xb6862750] 3: (_ULarm_step()+0x5b) [0xb67eecec] 4: (()+0x255e8) [0xb6cb05e8] 5: (GetStackTrace(void**, int, int)+0x25) [0xb6cb0a3e] 6: (tcmalloc::PageHeap::GrowHeap(unsigned int)+0xb9) [0xb6ca536a] 7: (tcmalloc::PageHeap::New(unsigned int)+0x79) [0xb6ca55e6] 8: (tcmalloc::CentralFreeList::Populate()+0x71) [0xb6ca45ce] 9: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x1b) [0xb6ca4760] 10: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x6d) [0xb6ca47de] 11: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, unsigned int)+0x51) [0xb6ca6a56] 12: (malloc()+0x22d) [0xb6cb1a8e] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mgr.node1.log --- end dump of recent events --- *Romain Raynaud*

1 0

mgr hangs with upmap balancer
by Bryan Stillwell 22 Nov '19

22 Nov '19

On multiple clusters we are seeing the mgr hang frequently when the balancer is enabled. It seems that the balancer is getting caught in some kind of infinite loop which chews up all the CPU for the mgr which causes problems with other modules like prometheus (we don't have the devicehealth module enabled yet). I've been able to reproduce the issue doing an offline balance as well using the osdmaptool: osdmaptool --debug-osd 10 osd.map --upmap balance-upmaps.sh --upmap-pool default.rgw.buckets.data --upmap-max 100 It seems to loop over the same group of PGs of ~7,000 PGs over and over again like this without finding any new upmaps that can be added: 2019-11-19 16:39:11.131518 7f85a156f300 10 trying 24.d91 2019-11-19 16:39:11.138035 7f85a156f300 10 trying 24.2e3c 2019-11-19 16:39:11.144162 7f85a156f300 10 trying 24.176b 2019-11-19 16:39:11.149671 7f85a156f300 10 trying 24.ac6 2019-11-19 16:39:11.155115 7f85a156f300 10 trying 24.2cb2 2019-11-19 16:39:11.160508 7f85a156f300 10 trying 24.129c 2019-11-19 16:39:11.166287 7f85a156f300 10 trying 24.181f 2019-11-19 16:39:11.171737 7f85a156f300 10 trying 24.3cb1 2019-11-19 16:39:11.177260 7f85a156f300 10 24.2177 already has pg_upmap_items [368,271] 2019-11-19 16:39:11.177268 7f85a156f300 10 trying 24.2177 2019-11-19 16:39:11.182590 7f85a156f300 10 trying 24.a4 2019-11-19 16:39:11.188053 7f85a156f300 10 trying 24.2583 2019-11-19 16:39:11.193545 7f85a156f300 10 24.93e already has pg_upmap_items [80,27] 2019-11-19 16:39:11.193553 7f85a156f300 10 trying 24.93e 2019-11-19 16:39:11.198858 7f85a156f300 10 trying 24.e67 2019-11-19 16:39:11.204224 7f85a156f300 10 trying 24.16d9 2019-11-19 16:39:11.209844 7f85a156f300 10 trying 24.11dc 2019-11-19 16:39:11.215303 7f85a156f300 10 trying 24.1f3d 2019-11-19 16:39:11.221074 7f85a156f300 10 trying 24.2a57 While this cluster is running Luminous (12.2.12), I've reproduced the loop using the same osdmap on Nautilus (14.2.4). Is there somewhere I can privately upload the osdmap for someone to troubleshoot the problem? Thanks, Bryan

2 2

Mimic (13.2.6) OSD daemon won't start up after system restart, with failed assert...
by aoanla＠gmail.com 21 Nov '19

21 Nov '19

Hi everyone, I'm looking for some advice on diagnosing an OSD issue. We have a Mimic cluster, not very full, with Bluestore OSDs. We recently had to bring the cluster down to allow power testing in the host datacentre, and when we brought things up again, 1 OSD daemon would not start. The log shows (cut to useful context): -314> 2019-11-21 15:55:15.561 7efdc049dd80 4 rocksdb: Options.ttl: 0 -314> 2019-11-21 15:55:15.563 7efdc049dd80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-000127 succeeded,manifest_file_number is 127, next_file_number is 264, last_sequence is 21956004, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 123 -314> 2019-11-21 15:55:15.563 7efdc049dd80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 255 -314> 2019-11-21 15:55:15.563 7efdc049dd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1574351715564768, "job": 1, "event": "recovery_started", "log_files": [252, 255]} -314> 2019-11-21 15:55:15.563 7efdc049dd80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/rocksdb/db/db_impl_open.cc:551] Recovering log #252 mode 0 -314> 2019-11-21 15:55:16.722 7efdc049dd80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/rocksdb/db/db_impl_open.cc:551] Recovering log #255 mode 0 -314> 2019-11-21 15:55:17.885 7efdc049dd80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/os/bluestore/KernelDevice.cc: In function 'virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IOContext*, bool)' thread 7efdc049dd80 time 2019-11-21 15:55:17.870632 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/os/bluestore/KernelDevice.cc: 825: FAILED assert((uint64_t)r == len) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7efdb788036b] 2: (()+0x26e4f7) [0x7efdb78804f7] 3: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4b4) [0x5619ab313144] 4: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x3c2) [0x5619ab2d59a2] 5: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x34) [0x5619ab2f88f4] 6: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x6b) [0x5619ab4e541b] 7: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xd8) [0x5619ab3f3148] 8: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x70) [0x5619ab3f3240] 9: (rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::string*, rocksdb::WALRecoveryMode)+0x12b) [0x5619ab3f351b] 10: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0xea2) [0x5619ab3a3bf2] 11: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x5619ab3a54e9] 12: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x5619ab3a6299] 13: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x5619ab3a7ac2] 14: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x5619ab27a43e] 15: (BlueStore::_open_db(bool, bool)+0xd6a) [0x5619ab205f9a] 16: (BlueStore::_mount(bool, bool)+0x4d1) [0x5619ab237071] 17: (OSD::init()+0x28f) [0x5619aaddeedf] 18: (main()+0x23a3) [0x5619aacbd7a3] 19: (__libc_start_main()+0xf5) [0x7efdb33f2505] The disk behind this OSD is very new, and hasn't been stressed very much, so I am not convinced it's a disk failure issue. Is this a known bug in Mimic (it's hard to find a similar bug in the bug tracker)... how should I diagnose this? Sam

1 0

Cephalocon 2020 will be March 4-5 in Seoul, South Korea!
by Sage Weil 20 Nov '19

20 Nov '19

Hi everyone, We're pleased to announce that the next Cephalocon will be March 3-5 in Seoul, South Korea! https://ceph.com/cephalocon/seoul-2020/ The CFP for the conference is now open: https://linuxfoundation.smapply.io/prog/cephalocon_2020 Main conference: March 4-5 Developer summit: March 3 Mark your calendars, and get your talk proposals in! The CFP will close in early December in order to get a final schedule published in early January. In addition to the two day conference, we will also have a developer summit on March 3 to take advantage of having so many developers in the same place at the same time. The developer sessions will include video conferencing so that remote developers will also be able to participate. A sponsorship prospectus will be available Real Soon Now. We hope you can join us!

1 0

scrub error on object storage pool
by M Ranga Swami Reddy 20 Nov '19

20 Nov '19

Hello - Recently we have upgraded to Luminous 12.2.11. After that we can see the scrub errors on the object storage pool only on daily basis. After repair, it will be cleared. But again it will come tomorrow after scrub performed the PG. Any known issue - on scrub errs with 12.2.11 version? Thanks Swami

1 0

msgr2 not used on OSDs in some Nautilus clusters
by Bryan Stillwell 19 Nov '19

19 Nov '19

I've upgraded 7 of our clusters to Nautilus (14.2.4) and noticed that on some of the clusters (3 out of 7) the OSDs aren't using msgr2 at all. Here's the output for osd.0 on 2 clusters of each type: ### Cluster 1 (v1 only): # ceph osd find 0 | jq -r '.addrs' { "addrvec": [ { "type": "v1", "addr": "10.26.0.33:6809", "nonce": 4185021 } ] } ### Cluster 2 (v1 only): # ceph osd find 0 | jq -r '.addrs' { "addrvec": [ { "type": "v1", "addr": "10.197.0.243:6801", "nonce": 3802140 } ] } ### Cluster 3 (v1 & v2): # ceph osd find 0 | jq -r '.addrs' { "addrvec": [ { "type": "v2", "addr": "10.32.0.36:6802", "nonce": 3167 }, { "type": "v1", "addr": "10.32.0.36:6804", "nonce": 3167 } ] } ### Cluster 4 (v1 & v2): # ceph osd find 0 | jq -r '.addrs' { "addrvec": [ { "type": "v2", "addr": "10.36.0.12:6820", "nonce": 3150 }, { "type": "v1", "addr": "10.36.0.12:6827", "nonce": 3150 } ] } All of the mon nodes have the same msgr settings of: # ceph daemon mon.$(hostname -s) config show | grep msgr "mon_warn_on_msgr2_not_enabled": "true", "ms_bind_msgr1": "true", "ms_bind_msgr2": "true", "ms_msgr2_encrypt_messages": "false", "ms_msgr2_sign_messages": "false", What could be causing this? All of the clusters are listening on port 3300 for v2 and 6789 for v1. I can even connect to port 3300 on the mon nodes from the OSD nodes. Thanks, Bryan

2 4

jewel OSDs refuse to start up again
by Janne Johansson 19 Nov '19

19 Nov '19

Three OSDs, holding the 3 replicas of a PG here are only half-starting, and hence that single PG gets stuck as "stale+active+clean". All died of suicide timeout while walking over a huge omap (pool 7 'default.rgw.buckets.index') and would not get the PG 7.b back online again. From the logs, they try to start normally, get into a bit of leveldb things, play the journal and then say nothing more. 2019-11-19 15:15:46.967543 7fe644fad840 0 set uid:gid to 167:167 (ceph:ceph) 2019-11-19 15:15:46.967600 7fe644fad840 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-osd, pid 5149 2019-11-19 15:15:47.026065 7fe644fad840 0 pidfile_write: ignore empty --pid-file 2019-11-19 15:15:47.078291 7fe644fad840 0 filestore(/var/lib/ceph/osd/ceph-22) backend xfs (magic 0x58465342) 2019-11-19 15:15:47.079317 7fe644fad840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2019-11-19 15:15:47.079331 7fe644fad840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option 2019-11-19 15:15:47.079352 7fe644fad840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_features: splice is supported 2019-11-19 15:15:47.080287 7fe644fad840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2019-11-19 15:15:47.080529 7fe644fad840 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_feature: extsize is disabled by conf 2019-11-19 15:15:47.095819 7fe644fad840 1 leveldb: Recovering log #2731809 2019-11-19 15:15:47.119792 7fe644fad840 1 leveldb: Level-0 table #2731812: started 2019-11-19 15:15:47.132107 7fe644fad840 1 leveldb: Level-0 table #2731812: 140642 bytes OK 2019-11-19 15:15:47.143782 7fe644fad840 1 leveldb: Delete type=0 #2731809 2019-11-19 15:15:47.147198 7fe644fad840 1 leveldb: Delete type=3 #2731792 2019-11-19 15:15:47.159339 7fe644fad840 0 filestore(/var/lib/ceph/osd/ceph-22) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2019-11-19 15:15:47.243262 7fe644fad840 1 journal _open /var/lib/ceph/osd/ceph-22/journal fd 18: 21472739328 bytes, block size 4096 bytes, directio = 1, aio = 1 At this point they consume a ton of cpu, systemd thinks all is fine, and this has been going on for some 5 hours. ceph -s think they are down, I can't talk to the OSDs remotely from a mon, but ceph daemon on the OSD hosts works normally, except I can't do anything from there except get conf or perf numbers. Strace shows they all keep looping over the same sequence: machine1: stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7fffd7c98080) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7fffd7c98080) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 machine2: stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7ffe0b664240) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7ffe0b664240) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 machine3: stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7ffc63518650) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7ffc63518650) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 Help wanted. -- May the most significant bit of your life be positive.

1 0

How proceed to change a crush rule and remap pg's?
by Maarten van Ingen 19 Nov '19

19 Nov '19

Hi, I have a small but impacting error in my crush rules. For unknown reasons the rules are not using host but osd to place the data and thus we have some nodes with all three copies instead of three different nodes. We noticed this when rebooting a node and a pg became stale. My crush rule: { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "osd" }, { "op": "emit" } ] }, Type should be host of course. And I want to alter this and move pg's such that all is as should. How can I best proceed in correcting this issue? I do like to throttle the remapping of the data so ceph itself won't be unavailable while the data is redistributed. We are running on Mimic (13.2.6), and this environment has been installed freshly as Mimic while using ceph-ansible. Current ceph -s output: cluster: id: <<fsid> health: HEALTH_OK services: mon: 3 daemons, quorum mon01,mon02,mon03 mgr: mon01(active), standbys: mon02, mon03 mds: cephfs-2/2/2 up {0=mon03=up:active,1=mon01=up:active}, 1 up:standby osd: 502 osds: 502 up, 502 in data: pools: 18 pools, 8192 pgs objects: 28.74 M objects, 100 TiB usage: 331 TiB used, 2.3 PiB / 2.6 PiB avail pgs: 8192 active+clean Cheers, Maarten van Ingen | Systems Expert | Distributed Data Processing | SURFsara | Science Park 140 | 1098 XG Amsterdam | | T +31 (0) 20 800 1300 | maarten.vaningen(a)surfsara.nl | https://surfsara.nl | We are ISO 27001 certified and meet the high requirements for information security.

2 2

corporate video production company in bangalore
by vhtnow11＠gmail.com 19 Nov '19

19 Nov '19

Best Video Production Company in Bangalore : We at VHTnow create visual masterpieces that engage, inspire and impact people's lives. Our services also include ad film and corporate film production in bangalore visit:https://vhtnow.com/

1 0

Re: Ssd cache question
by Wesley Peng 19 Nov '19

19 Nov '19

1 0

Ssd cache question
by Wesley Peng 18 Nov '19

18 Nov '19

2 1

Ceph manager causing MGR active switch
by Thomas Schneider 18 Nov '19

18 Nov '19

Hi, I can see the following error message regularely in MGR log: 2019-11-18 14:25:48.847 7fd9e6a3a700 0 mgr[dashboard] [18/Nov/2019:14:25:48] ENGINE Error in HTTPServer.tick Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start self.tick() File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick s, ssl_env = self.ssl_adapter.wrap(s) File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 67, in wrap server_side=True) File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket _context=self) File "/usr/lib/python2.7/ssl.py", line 599, in __init__ self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake self._sslobj.do_handshake() error: [Errno 0] Error 2019-11-18 14:25:49.027 7fd9e6a3a700 0 mgr[dashboard] [18/Nov/2019:14:25:49] ENGINE Error in HTTPServer.tick Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start self.tick() File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick s, ssl_env = self.ssl_adapter.wrap(s) File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 67, in wrap server_side=True) File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket _context=self) File "/usr/lib/python2.7/ssl.py", line 599, in __init__ self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake self._sslobj.do_handshake() SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:727) In many cases this error causes a switch of the active MGR node. But there's another impact on the Ceph Dashboard directly that hangs completely when this error is logged. Any advise how to fix this issue is appreciated. THX

1 0

nfs ganesha rgw write errors
by Marc Roos 18 Nov '19

18 Nov '19

Hi Daniel, I am able to mount the buckets with your config, however when I try to write something, my logs get a lot of these errors: svc_732] nfs4_Errno_verbose :NFS4 :CRIT :Error I/O error in nfs4_write_cb converted to NFS4ERR_IO but was set non-retryable Any chance you know how to resolve this?

2 1

Nfs-ganesha rpm still has samba package dependency
by Marc Roos 18 Nov '19

18 Nov '19

======================================================================== ================================================== Package Arch Version Repository Size ======================================================================== ================================================== Installing: nfs-ganesha x86_64 2.8.1.2-0.1.el7 CentOS7-custom 680 k nfs-ganesha-ceph x86_64 2.8.1.2-0.1.el7 CentOS7-custom 30 k nfs-ganesha-mem x86_64 2.8.1.2-0.1.el7 CentOS7-custom 30 k nfs-ganesha-rgw x86_64 2.8.1.2-0.1.el7 CentOS7-custom 21 k nfs-ganesha-vfs x86_64 2.8.1.2-0.1.el7 CentOS7-custom 44 k nfs-ganesha-xfs x86_64 2.8.1.2-0.1.el7 CentOS7-custom 42 k Installing for dependencies: libldb x86_64 1.4.2-1.el7 CentOS7 144 k libntirpc x86_64 1.8.0-0.1.el7 CentOS7-custom 113 k libtalloc x86_64 2.1.14-1.el7 CentOS7 32 k libtdb x86_64 1.3.16-1.el7 CentOS7 48 k libtevent x86_64 0.9.37-1.el7 CentOS7 40 k libwbclient x86_64 4.9.1-6.el7 CentOS7 110 k samba-client-libs x86_64 4.9.1-6.el7 CentOS7 4.9 M samba-common noarch 4.9.1-6.el7 CentOS7 209 k samba-common-libs x86_64 4.9.1-6.el7 CentOS7 170 k Transaction Summary ======================================================================== ================================================== Install 6 Packages (+9 Dependent packages) Total download size: 6.6 M Installed size: 23 M

2 1

Full FLash NVME Cluster recommendation
by Yoann Moulin 18 Nov '19

18 Nov '19

Hello, I'm going to deploy a new cluster soon based on 6.4TB NVME PCI-E Cards, I will have only 1 NVME card per node and 38 nodes. The use case is to offer cephfs volumes for a k8s platform, I plan to use an EC-POOL 8+3 for the cephfs_data pool. Do you have recommendations for the setup or mistakes to avoid? I use ceph-ansible to deploy all myclusters. Best regards, -- Yoann Moulin EPFL IC-IT

3 3

Re: Full FLash NVME Cluster recommendation
by Yoann Moulin 18 Nov '19

18 Nov '19

Hello Nathan, >>>> I'm going to deploy a new cluster soon based on 6.4TB NVME PCI-E Cards, I will have only 1 NVME card per node and 38 nodes. >>>> >>>> The use case is to offer cephfs volumes for a k8s platform, I plan to use an EC-POOL 8+3 for the cephfs_data pool. >>>> >>>> Do you have recommendations for the setup or mistakes to avoid? I use ceph-ansible to deploy all myclusters. >>> >>> In order to get optimal performance out of NVMe, you will want very >>> fast cores, and you will probably have to split each NVMe card into >>> 2-4 OSD partitions in order to throw enough cores at it. That's a good idea ! If I have enough time, I'll try to do some benchmark with 2 and 4 OSD partitions. >> I’ve been trying unsuccessfully to convince some folks of the need for fast cores, there’s the idea that the effect would be slight. Do >> you have any numbers? I’ve also read a claim that each BlueStore will use 3-4 cores, They’re listening to me though about splitting the >> card into multiple OSDs. > > Bluestore will use about 4 cores, but in my experience, the maximum > utilization I've seen has been something like: 100%, 100%, 50%, 50% > > So those first 2 cores are the bottleneck for pure OSD IOPS. This sort > of pattern isn't uncommon in multithreaded programs. This was on HDD > OSDs with DB/WAL on NVMe, as well as some small metadata OSDs on pure > NVMe. SSD OSDs default to 2 threads per shard, and HDD to 1, but we > had to set HDD to 2 as well when we enabled NVMe WAL/DB. Otherwise the > OSDs ran out of CPU and failed to heartbeat when under load. I believe > that if we had 50% faster cores, we might not have needed to do this. > > On SSDs/NVMe you can compensate for slower cores with more OSDs, but > of course only for parallel operations. Anything that is > serial+synchronous, not so much. I would expect something like 4 OSDs > per NVMe, 4 cores per OSD. That's already 16 cores per node just for > OSDs. > > Our bottleneck in practice is the Ceph MDS, which seems to use exactly > 2 cores and has no setting to change this. As far as I can tell, if we > had 50% faster cores just for the MDS, I would expect roughly +50% > performance in terms of metadata ops/second. Each filesystem has it's > own rank-0 MDS, so this load will be split across daemons. The MDS can > also use a ton of RAM (32GB) if the clients have a working set of 1 > million+ files. Multi-mds exists to further split the load, but is > quite new and I would not trust it. CephFS in general is likely where > you will have the most issues, as it both new and complex compared to > a simple object store. Having an MDS in standby-replay mode keeps it's > RAM cache synced with the active, so you get far faster failover ( > O(seconds) rather than O(minutes) with a few million file caps) but > you use the same RAM again. > > So, IMHO, you will want at least: > CPU: > 16 cores per 1-card NVMe OSD node. 2 cores per filesystem (maybe 1 if > you don't expect a lot of simultaneous load?) > > RAM: > The Bluestore default is 4GB per OSD, so 16GB per node. > ~32GB of RAM per active and standby-replay MDS if you expect file > counts in the millions, so 64GB per filesystem. The context is 3 Intel Server 1U for MONs/MDSs/MGRs services + K8s daemons CPU : 2 x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (24c/48t) Memory : 64GB Disk OS : 2x Intel SSD DC S3520 240GB 38 Dell C4140 1U for OSD nodes : CPU : 2 x Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz (28c/56t) Memory : 384GB GPU : 4 Nvidia V100 32GB NVLink Disk OS : M.2 240G NVME : Dell 6.4TB NVME PCI-E Drive (Samsung PM1725b), only 1 slot available Each server is used in a k8s cluster to give access to GPUs and CPUs for X-learning labs. Ceph have to share the CPU and memory with the compute K8s cluster. > 128GB of RAM per node ought to do, if you have less than 14 filesystems? I plan to have only 1 filesystem. Thanks to all those useful information. Best regards, -- Yoann Moulin EPFL IC-IT

1 0

PG in state: creating+down
by Thomas Schneider 18 Nov '19

18 Nov '19

Hi, ceph health is reporting: pg 59.1c is creating+down, acting [426,438] root@ld3955:~# ceph health detail HEALTH_WARN 1 MDSs report slow metadata IOs; noscrub,nodeep-scrub flag(s) set; Reduced data availability: 1 pg inactive, 1 pg down; 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio; mons ld5505,ld5506 are low on available space MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs mdsld4465(mds.0): 8 slow metadata IOs are blocked > 30 secs, oldest blocked for 120721 secs OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg down pg 59.1c is creating+down, acting [426,438] MON_DISK_LOW mons ld5505,ld5506 are low on available space mon.ld5505 has 22% avail mon.ld5506 has 29% avail root@ld3955:~# ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 59.1c creating+down [426,438] 426 [426,438] 426 How can I fix this? THX

3 4

Cannot list RBDs in any pool / cannot mount any RBD
by Thomas Schneider 15 Nov '19

15 Nov '19

Hi, when I execute this command rbd ls -l <pool-name> to list all RBDs I get spamming errors: 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1 Sender did not set CEPH_MSG_FOOTER_SIGNED. 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1 Message signature does not match contents. 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1Signature on message: 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1 sig: 0 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1Locally calculated signature: 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1 sig_check:6229148783016323662 2019-11-15 11:29:19.428 7fd852678700 0 Signature failed. 2019-11-15 11:29:19.428 7fd852678700 0 --1- 10.97.206.91:0/2841811017 >> v1:10.97.206.97:6884/265976 conn(0x7fd834090770 0x7fd83408c190 :-1 s=READ_FOOTER_AND_DISPATCH pgs=42068 cs=1 l=1).handle_message_footer Signature check failed 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1 Sender did not set CEPH_MSG_FOOTER_SIGNED. 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1 Message signature does not match contents. 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1Signature on message: 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1 sig: 0 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1Locally calculated signature: 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1 sig_check:4639667068846516939 2019-11-15 11:29:19.428 7fd852678700 0 Signature failed. 2019-11-15 11:29:19.428 7fd852678700 0 --1- 10.97.206.91:0/2841811017 >> v1:10.97.206.97:6884/265976 conn(0x7fd83408c990 0x7fd83408b160 :-1 s=READ_FOOTER_AND_DISPATCH pgs=42069 cs=1 l=1).handle_message_footer Signature check failed 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1 Sender did not set CEPH_MSG_FOOTER_SIGNED. 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1 Message signature does not match contents. 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1Signature on message: 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1 sig: 0 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1Locally calculated signature: 2019-11-15 11:29:19.428 7fd852678700 0 SIGN: MSG 1 sig_check:12754808375040063976 2019-11-15 11:29:19.428 7fd852678700 0 Signature failed. 2019-11-15 11:29:19.428 7fd852678700 0 --1- 10.97.206.91:0/2841811017 >> v1:10.97.206.97:6884/265976 conn(0x7fd834090770 0x7fd83408c190 :-1 s=READ_FOOTER_AND_DISPATCH pgs=42070 cs=1 l=1).handle_message_footer Signature check failed I allready stopped and started all MON services w/o success. This is the output of config dump: root@ld3955:~# ceph config-key dump | grep config/ "config/mgr/mgr/balancer/active": "true", "config/mgr/mgr/balancer/mode": "upmap", "config/mgr/mgr/balancer/pool_ids": "11,59,60,61", "config/mgr/mgr/balancer/upmap_max_iterations": "2", "config/mgr/mgr/dashboard/url_prefix": "dashboard", "config/mgr/mgr/devicehealth/enable_monitoring": "false", "config/mgr/target_max_misplaced_ratio": "0.010000", And this is the config of an OSD showing that cephx authentication is activated: root@ld3955:~# ceph config show osd.0 NAME VALUE SOURCE OVERRIDES IGNORES auth_client_required cephx file auth_cluster_required cephx file auth_service_required cephx file bluestore_block_db_size 10737418240 file cephx_cluster_require_signatures false file cephx_require_signatures false file cephx_sign_messages false file cluster_network 192.168.1.0/27 file daemonize false override debug_ms 0/0 file keyring $osd_data/keyring default leveldb_log default mon_allow_pool_delete true file mon_host 10.97.206.93 10.97.206.94 10.97.206.95 file mon_osd_full_ratio 0.850000 file mon_osd_nearfull_ratio 0.750000 file osd_crush_update_on_start false file osd_deep_scrub_interval 1209600.000000 file osd_journal_size 1024 file osd_max_backfills 2 file osd_op_queue wpq file osd_op_queue_cut_off high file osd_pool_default_min_size 2 file osd_pool_default_size 3 file osd_scrub_begin_hour 21 file osd_scrub_end_hour 8 file osd_scrub_sleep 0.100000 file public_network 10.97.206.0/24 file rbd_default_features 61 default setgroup ceph cmdline setuser ceph cmdline How can I fix this error? THX

3 4

mds can't trim journal
by locallocal 15 Nov '19

15 Nov '19

hi,cool guys, Recently,we had encountered a problem,the journal of MDS daemon could't be trimmed,resulting in a large amount of space occupied by the metadata pool.so what we could think out was using the admin socket command to flush journal,you know,It got worse,the admin thread of MDS was also stuck and after that we couldn't configure the log level.After analyzing the code, we found some segments couldn't get out out from expiring queue.but we don't know why and where is stuck in the function void(LogSegment::try_to_expire(MDSRank *mds, MDSGatherBuilder &gather_bld, int op_prio). any ideas or advice?Thanks a lot.Here are some cluster information: Version: luminous(v12.2.12) MDS debug log: 5 mds.0.log trim already expiring segment 3658103659/11516554553473, 980 events 5 mds.0.log trim already expiring segment 3658104639/11516556356904, 1024 events 5 mds.0.log trim already expiring segment 3658105663/11516558241475, 1024 events cephfs-journal-tool: { "magic": "ceph fs volume v011", "write_pos": 11836049063598, "expire_pos": 11516554553473, "trimmed_pos": 11516552151040, "stream_format": 1, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 2, "pool_ns": "" } } | | locallocal | | locallocal(a)163.com | 签名由网易邮箱大师定制

1 0

Rolling out radosgw-admin4j v2.0.2
by hrchu 15 Nov '19

15 Nov '19

radosgw-admin4j is an admin client in Java that allows provisioning and control of Ceph object store. In version 2.0.2, Java 11 and Ceph Nautilus are supported. See https://github.com/twonote/radosgw-admin4j for more details.

1 0

Can't Add Zone at Remote Multisite Cluster
by Mac Wynkoop 14 Nov '19

14 Nov '19

Hi All, So, I am trying to create a site-specifc zonegroup at my 2nd site's Ceph cluster. Upon creating the zonegroup and a placeholder master zone at my master site, I go to do a period update and commit, and this is what it returns to me: (hostname) ~ $ radosgw-admin period commit 2019-11-14 22:27:41.023 7f87e87359c0 1 Cannot find zone id=c7403ff7-d141-46b1-a185-67edb10c23e1 (name=hou-placeholder-zone), switching to local zonegroup configuration 2019-11-14 22:27:41.023 7f87e87359c0 -1 Cannot find zone id=c7403ff7-d141-46b1-a185-67edb10c23e1 (name=hou-placeholder-zone) 2019-11-14 22:27:41.023 7f87e87359c0 0 ERROR: failed to start notify service ((22) Invalid argument 2019-11-14 22:27:41.023 7f87e87359c0 0 ERROR: failed to init services (ret=(22) Invalid argument) couldn't init storage provider (hostname) ~ $ radosgw-admin zone list { "default_info": "c7403ff7-d141-46b1-a185-67edb10c23e1", "zones": [ "hou-placeholder-zone", "hou", "hou-ec-1" ] } (hostname)~ $ It's like it doesn't see that the zone actually exists, and won't let me commit the period. Any ideas as to what's going on here? Thanks, Mac

1 0

Bad links on ceph.io for mailing lists
by Bryan Stillwell 14 Nov '19

14 Nov '19

There are some bad links to the mailing list subscribe/unsubscribe/archives on this page that should get updated: https://ceph.io/resources/ The subscribe/unsubscribe/archives links point to the old lists vger and lists.ceph.com, and not the new lists on lists.ceph.io: ceph-devel subscribe => mailto:majordomo@vger.kernel.org?body=subscribe ceph-devel unsubscribe => mailto:majordomo@vger.kernel.org?body=unsubscribe ceph-devel archives => http://marc.info/?l=ceph-devel ceph-users subscribe => mailto:ceph-users-join@lists.ceph.com unsubscribe => mailto:ceph-users-leave@lists.ceph.com archives => http://lists.ceph.com/pipermail/ceph-users-ceph.com/ Bryan

3 2

Re: osdmaps not trimmed until ceph-mon's restarted (if cluster has a down osd)
by Dan van der Ster 14 Nov '19

14 Nov '19

On Thursday, November 14, 2019, Nathan Cutler <ncutler(a)suse.com> wrote: > > Hi Dan: > > > I might have found the reason why several of our clusters (and maybe > > Bryan's too) are getting stuck not trimming osdmaps. > > It seems that when an osd fails, the min_last_epoch_clean gets stuck > > forever (even long after HEALTH_OK), until the ceph-mons are > > restarted. > > > > I've updated the ticket: https://tracker.ceph.com/issues/41154 > > Did you mean to write https://tracker.ceph.com/issues/37875 here? Oops yes that's it. Not sure where that other link came from. Thanks! Dan > Nathan

1 0

increasing PG count - limiting disruption
by Frank R 14 Nov '19

14 Nov '19

Hi all, When increasing the number of placement groups for a pool by a large amount (say 2048 to 4096) is it better to go in small steps or all at once? This is a filestore cluster. Thanks, Frank

2 1

mds crash loop - cephfs disaster recovery
by Karsten Nielsen 14 Nov '19

14 Nov '19

I am a problem with my mds that is in a crash loop, with the help of Yan, Zheng I have run a few attempts to save it but it seems that it is not going the way it should. I am reading through this documentation. https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ If I use the last step to get it back by creating an alternative metadata pool for recovery. If that works how do I remove the broken metadata pool and rename the new one to match the broken one ? Thanks, - Karsten

1 0

Adding new non-containerised hosts to current contanerised environment and moving away from containers forward
by Jeremi Avenant 14 Nov '19

14 Nov '19

Good day We currently have 12 nodes in 4 Racks (3x4) and getting another 3 nodes to complete the 5th rack on Version 12.2.12, using ceph-ansible & docker containers. With the 3 new nodes (1 rack bucket) we would like to make use of a non-containerised setup since our long term plan is to completely move away from OSD containers. How would one go forward running a hybrid environment in the interim until we've rebuilt all our existing osd nodes? I can assume that i would require a separate /ceph-ansible directory with site.yml to my existing site-docker.yml Would I require separate inventory files, one for the existing 12 nodes & one for the 3 new nodes? How would I connect the new nodes to the current controller (mds,mgr, mon) container nodes? Regards -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za <http://www.uwc.ac.za/> E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

2 1

Allowing cephfs clients to reconnect
by Mikael Öhman 14 Nov '19

14 Nov '19

Hi, I'm trying to make our system a bit more fault tolerant, and I struggle a bit with letting clients reconnect if they have lost contact for a while. When there is a temporary network problem, I would like clients to block I/O, wait for a connection, and resume. Do I have any options other than just increasing mds_session_autoclose ? Is there a downside for using very large value here (like, a full day?)? I expect all clients to be connected at all times anyway when things are running normally. What I see right now (if the disconnect is sufficiently long) is that the ceph client releases the I/O block, and you get permission denied on all I/O operations on the existing mount point. Re-mounting it works, but, this also requires killing off all active session blocking unmounting. Basically, just overall bad is this happens, and I would prefer almost any other option. I can see that the client tries a reconnect when this happens: Nov 12 11:53:24 hebbe01-3 kernel: libceph: mds0 10.43.20.3:6800 connection reset Nov 12 11:53:24 hebbe01-3 kernel: libceph: reset on mds0 Nov 12 11:53:24 hebbe01-3 kernel: ceph: mds0 closed our session Nov 12 11:53:24 hebbe01-3 kernel: ceph: mds0 reconnect start Nov 12 11:53:24 hebbe01-3 kernel: ceph: mds0 reconnect denied Nov 12 11:56:55 hebbe01-3 kernel: libceph: mds0 10.43.20.3:6800 socket closed (con state NEGOTIATING) Nov 12 11:56:55 hebbe01-3 kernel: ceph: mds0 rejected session but the logs on the MDS server disallows it as it's not in a "reconnect state"- So, if I understand this correctly, reconnecting is just available in the case that the MDS server was rebooted? Best regards, Mikael

2 1

Re: custom x-amz-request-id
by Robin H. Johnson 13 Nov '19

13 Nov '19

On Wed, Nov 13, 2019 at 04:32:45PM +0000, Arash Shams wrote: > Hi everybody > Im using Nginx in front of radosgw and I generate the request id header on nginx can I pass the same value to radosgw and tell it use this header instead of generating a new one ? > > nginx sample : > > more_set_input_headers "x-amz-request-id: $txid" You'll have to patch RGW's source to do this: 1. Copy the header from request to response. 2. Do NOT store the header as metadata in PutObject. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2(a)gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

1 0

Counting OSD maps
by Bryan Stillwell 13 Nov '19

13 Nov '19

With FileStore you can get the number of OSD maps for an OSD by using a simple find command: # rpm -q ceph ceph-12.2.12-0.el7.x86_64 # find /var/lib/ceph/osd/ceph-420/current/meta/ -name 'osdmap*' | wc -l 42486 Does anyone know of an equivalent command that can be used with BlueStore? Thanks, Bryan

2 1

custom x-amz-request-id
by Arash Shams 13 Nov '19

13 Nov '19

Hi everybody Im using Nginx in front of radosgw and I generate the request id header on nginx can I pass the same value to radosgw and tell it use this header instead of generating a new one ? nginx sample : more_set_input_headers "x-amz-request-id: $txid" Thanks

1 0

Ceph I/O issues on all SSD cluster
by Dennis Højgaard | Powerhosting Support 13 Nov '19

13 Nov '19

OK, where to start. I have been debugging intensively the last two days, but can't seem to wrap my head around the performance issues we see in one of our two hyperconverged (ceph) proxmox clusters. Let me introduce our two clusters and some of the debugging results. *1. Cluster for internal purposes (performs as expected)* 3 x Supermicro servers with identical specs: CPU: 1 x 7-7700K CPU @ 4.20GHz (1 Socket) 4 cores / 8 threads RAM: 64 GB RAM OSDs: 4 per node. 1 per SSD (Intel S4610) (12 OSDs in all) 1 x 10GbE RJ45 nic. MTU 9000 No bonding _A total of 3 servers with a total of 12 OSDs_ Network: 1 x Unifi Switch 16 XG *2. Cluster for VPS's for customers (performs much worse than internal)* 3 x Dell R630 with the following specs: CPU: 2 x E5-2697 v3 @ 2.60GHz (2 Sockets) 28 cores / 56 threads RAM 256GB OSDs: 10 per node. 1 per SSD (Intel S4610) 1 x 10GbE SFP+ nic with 2 ports bonded via LACP (bond-xmit-hash-policy layer3+4). MTU 9000 2 x Supermicro X11SRM-VF with the following specs: CPU 1 x 1 W-2145 CPU @ 3.70GHz (1 Socket) 8 cores / 16 threads RAM: 256 GB OSDs 8 per node. 1 per SSD (Intel S4610) 1 x 10GbE SFP+ nic with 2 ports bonded via LACP (bond-xmit-hash-policy layer3+4). MTU 9000 1 x Dell R630 with the following specs: CPU 2 x CPU E5-2696 v4 @ 2.20GHz (2 Sockets) 44 cores / 88 threads RAM: 256 GB OSDs 8 per node. 1 per SSD (Intel S4610) 1 x 10GbE SFP+ nic with 2 ports bonded via LACP (bond-xmit-hash-policy layer3+4). MTU 9000 _A total of 6 servers with a total of 54 OSDs_ Network: 2 x Dell N4032F 10GbE SFP+ Switch connected with MLAG. Each node is connected to each switch. To get a fair comparison i made the following fio tests one one host in each cluster on a rbd block device that i created: *1. cluster:* |fio --randrepeat=1 --ioengine=libaio --sync=1 --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.12 Starting 1 process test: Laying out IO file (1 file / 4096MiB) Jobs: 1 (f=1): [w(1)][100.0%][w=25.0MiB/s][w=6409 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=869177: Wed Nov 13 11:15:59 2019 write: IOPS=4158, BW=16.2MiB/s (17.0MB/s)(4096MiB/252126msec); 0 zone resets bw ( KiB/s): min= 2075, max=32968, per=99.96%, avg=16627.60, stdev=9635.42, samples=504 iops : min= 518, max= 8242, avg=4156.88, stdev=2408.86, samples=504 cpu : usr=0.53%, sys=3.81%, ctx=109599, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=16.2MiB/s (17.0MB/s), 16.2MiB/s-16.2MiB/s (17.0MB/s-17.0MB/s), io=4096MiB (4295MB), run=252126-252126msec Disk stats (read/write): rbd0: ios=46/1221898, merge=0/1870438, ticks=25/4654920, in_queue=1980016, util=84.70%| *2. cluster* |fio --randrepeat=1 --ioengine=libaio --sync=1 --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.12 Starting 1 process Jobs: 1 (f=1): [w(1)][99.9%][w=7024KiB/s][w=1756 IOPS][eta 00m:01s] test: (groupid=0, jobs=1): err= 0: pid=794096: Wed Nov 13 11:25:56 2019 write: IOPS=1353, BW=5415KiB/s (5545kB/s)(4096MiB/774601msec); 0 zone resets bw ( KiB/s): min= 40, max=30600, per=100.00%, avg=5420.24, stdev=3710.17, samples=1547 iops : min= 10, max= 7650, avg=1355.06, stdev=927.54, samples=1547 cpu : usr=0.16%, sys=1.19%, ctx=100028, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=5415KiB/s (5545kB/s), 5415KiB/s-5415KiB/s (5545kB/s-5545kB/s), io=4096MiB (4295MB), run=774601-774601msec Disk stats (read/write): rbd0: ios=0/1222639, merge=0/1784089, ticks=0/12124812, in_queue=9514280, util=45.14%| And identical rados bench tests: 1. cluster:https://i.imgur.com/AdARCA6.png 2. clusterhttps://i.imgur.com/Di7mYQh.png I have fio tested all disks. I have tested the network. I can't seem to find the reason why performance on my 2. cluster is relatively poor compared to 1. cluster. -- Dennis Højgaard Powerhosting Support *t:* +45 7222 4457 |*e:* dh(a)powerhosting.dk |*w:* https://powerhosting.dk

1 0

xattrs on snapshots
by Toby Darling 13 Nov '19

13 Nov '19

Hi Folks I'm writing a backup/sync process for our ceph cluster. The process takes a snapshot of the system that's being backed up and rsync's from that to another ceph cluster I was hoping to use the snapshot xattrs to verify a successful backup by comparing ceph.dir.rbytes and ceph.dir.rentries between the snapshot and backup target. However, while files added to the source directory after the snapshot is taken don't appear in the snapshot, ceph.dir.rbytes and ceph.dir.rctime are updated in the xattrs of both. $ fallocate -l 1K test1 $ getfattr -n ceph.dir.rbytes . # file: . ceph.dir.rbytes="1024" $ mkdir .snap/testsnap $ fallocate -l 2K test2 $ getfattr -n ceph.dir.rbytes . .snap/testsnap # file: . ceph.dir.rbytes="3072" # file: .snap/testsnap ceph.dir.rbytes="3072" $ getfattr -n ceph.dir.rctime . .snap/testsnap # file: . ceph.dir.rctime="1573558448.09736131125" # file: .snap/testsnap ceph.dir.rctime="1573558448.09736131125" $ tree -a --si . .snap . |-- [1.0k] test1 `-- [2.0k] test2 .snap `-- [3.1k] testsnap `-- [1.0k] test1 1 directory, 3 files Have I misunderstood? Is this expected behaviour? Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology Francis Crick Avenue Cambridge Biomedical Campus Cambridge CB2 0QH Phone 01223 267070

3 3

ceph clients and cluster map
by Frank R 12 Nov '19

12 Nov '19

Hi all, When the cluster map changes say due to a failed OSD how are the clients with established sessions notified? thx Frank

1 0

Invisible braces
by bluetoothdental123＠gmail.com 12 Nov '19

12 Nov '19

https://www.bluetoothdentalclinic.co.in/invisalign Bluetooth dental clinic provides the cheapest invisible braces or invisalign in bangalore and the best clinic to get clear braces in india.

1 0

Re: mds crash loop
by Karsten Nielsen 12 Nov '19

12 Nov '19

-----Original message----- From: Karsten Nielsen <karsten(a)foo-bar.dk> Sent: Tue 12-11-2019 10:30 Subject: [ceph-users] Re: mds crash loop To: Yan, Zheng <ukernel(a)gmail.com>; CC: ceph-users(a)ceph.io; > -----Original message----- > From: Yan, Zheng <ukernel(a)gmail.com> > Sent: Mon 11-11-2019 15:09 > Subject: Re: [ceph-users] Re: mds crash loop > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > CC: ceph-users(a)ceph.io; > > On Mon, Nov 11, 2019 at 5:09 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > I started a job that moved some files around in the cephfs cluster that > > resulted in the mds to go back into the crash loop. > > > Logs are here: > > > http://s3.foo-bar.dk/mds-dumps/mds.log-20191111 > > > > > > Any help would be appriciated. > > > > > > > looks like snaptable is corrupted. > > > > nautilus version (14.2.2) of ‘cephfs-data-scan scan_links’ can fix > > snaptable. hopefully it will fix your issue. > > > > you don't need to upgrade whole cluster. Just install nautilus in a > > temp machine or compile ceph from source. > > I did run the command that you suggested, it did not unfortunately fix the > problem. > > http://s3.foo-bar.dk/mds-dumps/mds.log-20191112 > The output from the command is this: sudo docker exec -it rgw2 cephfs-data-scan scan_links 2019-11-12 08:46:27.025 7fe775dd7d80 -1 datascan.scan_links: Remove duplicated ino 0x0x100026d17d4 from 0x100013b0d3d/latest.log 2019-11-12 08:46:28.665 7fe775dd7d80 -1 datascan.load_table: unable to read mds table 'mds1_inotable': (2) No such file or directory 2019-11-12 08:46:28.665 7fe775dd7d80 -1 mds.1.inotable: erasing 0x20000000000 to 0x2000000d665 2019-11-12 08:46:28.793 7fe775dd7d80 -1 datascan.load_table: unable to read mds table 'mds2_inotable': (2) No such file or directory 2019-11-12 08:46:28.793 7fe775dd7d80 -1 mds.2.inotable: erasing 0x30000000000 to 0x300000228f5 2019-11-12 08:46:29.345 7fe775dd7d80 -1 mds.0.snap updating last_snap 1 -> 3 > > > > > > > > > > > - Karsten > > > > > > -----Original message----- > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > Sent: Thu 07-11-2019 14:20 > > > Subject: Re: [ceph-users] Re: mds crash loop > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > CC: ceph-users(a)ceph.io; > > > > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > > > > > That is awesome. > > > > > > > > > > Now I just need to figure out where the lost+found files needs to go. > > > > > And what happened to the missing objects for the dirs. > > > > > > > > > > > > > lost+found files are likely files that were deleted. you can keep the > > > > lost+found dir for a while, then delete the 'lost+found' directory. > > > > > > > > for 'missing object' dirs, mv all of them to a temp directory, such as > > > > /mnt/cephfs/missing_obj_dirs. > > > > Then run command 'ceph daemon mds.x scrub_patch /missing_obj_dirs > > > > force recursive repair'. wait a minute, the rm -rf > > > > /mnt/cephfs/missing_obj_dirs > > > > > > > > > Any tool that is able to do that ? > > > > > > > > > > Thanks > > > > > - Karsten > > > > > > > > > > -----Original message----- > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > Sent: Thu 07-11-2019 09:22 > > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > CC: ceph-users(a)ceph.io; > > > > > > I have tracked down the root cause. See > > > > https://tracker.ceph.com/issues/42675 > > > > > > > > > > > > Regards > > > > > > Yan, Zheng > > > > > > > > > > > > On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen <karsten(a)foo-bar.dk> > > wrote: > > > > > > > > > > > > > > -----Original message----- > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > Sent: Thu 07-11-2019 07:21 > > > > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen > <karsten(a)foo-bar.dk> > > > > wrote: > > > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > > Sent: Wed 06-11-2019 14:16 > > > > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen > > <karsten(a)foo-bar.dk> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > > > > Sent: Wed 06-11-2019 08:15 > > > > > > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen > > > > <karsten(a)foo-bar.dk> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > Last week I upgraded my ceph cluster from luminus to > mimic > > > > 13.2.6 > > > > > > > > > > > > > It was running fine for a while but yesterday my mds > went > > > > into a > > > > > > crash > > > > > > > > > > loop. > > > > > > > > > > > > > > > > > > > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of > > which > > > > is > > > > > > > > running > > > > > > > > > > the > > > > > > > > > > > > same crash loop. > > > > > > > > > > > > > I am running ceph based on > > > > https://hub.docker.com/r/ceph/daemon > > > > > > > > version > > > > > > > > > > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv > > store. > > > > > > > > > > > > > > > > > > > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > > > > > > > > > > > > > > > > > > > Yan, Zheng > > > > > > > > > > > > > > > > > > > > > > Yes I have set that and had to move to pastebin.com as > debian > > > > > > apperently > > > > > > > > only > > > > > > > > > > supports 150k > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://pastebin.com/Gv7c5h54 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looks like on-disk root inode is corrupted. have you > > encountered any > > > > > > > > > > unusually things during the upgrade? > > > > > > > > > > > > > > > > > > > > please run 'rados -p <cephfs metadata pool> stat > > 1.00000000.inode' , > > > > > > > > > > check if the object is modified before or after the 'luminous > -> > > > > > > > > > > 13.2.6' upgrade. > > > > > > > > > > To fix the corrupted object. Run 'cephfs-data-scan init > > > > > > > > > > --force-init'. Then restart mds. After mds become active, run > > 'ceph > > > > > > > > > > daemon mds.x scrub_path / force repair' > > > > > > > > > > > > > > > > > > > > > > > > > > > > I followed the steps I got the mds started but now a lot of > files > > are > > > > in > > > > > > > > lost+found 24283 and I have these errors in the mds log > > > > > > > > > > > > > > > > > 'cephfs-data-scan init --force-init' does not move files into > > > > > > > > lost+found. have you ever run other 'cephfs-data-scan foo' > command > > or > > > > > > > > 'cephfs-journal-tool foo' command? > > > > > > > > > > > > > > I have had a similar problem with the cluster before where I went > > through > > > > the > > > > > > cycle of: > > > > > > > https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ -> Using > an > > > > > > alternate metadata pool for recovery > > > > > > > > > > > > > > I did run the cephfs-journal-tool journal reset command, mostly > > because > > > > > > cephfs is not that utilized so I thought it was safe to do as after > the > > > > upgrade > > > > > > the cluster has not been used much, so data lose would be minimal - > > > > apparently > > > > > > I was wrong. > > > > > > > > > > > > > > > > > > > > > > > > 2019-11-06 20:20:18.215 7f0bd9090700 1 mds.0.32011 cluster > > recovered. > > > > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 0 > > mds.0.cache.dir(0x100013acfcb) > > > > > > > > _fetched missing object for [dir 0x100013acfcb > > > > > > /nextcloud/custom_apps/carnet/ > > > > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() > n() > > > > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100] > > > > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 log_channel(cluster) log > > > > [ERR] : > > > > > > dir > > > > > > > > 0x100013acfcb object missing on disk; some files may be lost > > > > > > > > (/nextcloud/custom_apps/carnet) > > > > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 0 > > mds.0.cache.dir(0x100013a3156) > > > > > > > > _fetched missing object for [dir 0x100013a3156 > > > > /nextcloud/custom_apps/mail/ > > > > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() > n() > > > > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000] > > > > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 log_channel(cluster) log > > > > [ERR] : > > > > > > dir > > > > > > > > 0x100013a3156 object missing on disk; some files may be lost > > > > > > > > (/nextcloud/custom_apps/mail) > > > > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 0 > > mds.0.cache.dir(0x100013abb3c) > > > > > > > > _fetched missing object for [dir 0x100013abb3c > > > > > > > > /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 > ap=1+0+0 > > > > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 > authpin=1 > > > > > > > > 0x55d4dcc40700] > > > > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 log_channel(cluster) log > > > > [ERR] : > > > > > > dir > > > > > > > > 0x100013abb3c object missing on disk; some files may be lost > > > > > > > > (/nextcloud/custom_apps/passwords) > > > > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 0 > > mds.0.cache.dir(0x100013a9b9b) > > > > > > > > _fetched missing object for [dir 0x100013a9b9b > > > > > > > > /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 > ap=1+0+0 > > > > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 > authpin=1 > > > > > > > > 0x55d4dcc40e00] > > > > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 log_channel(cluster) log > > > > [ERR] : > > > > > > dir > > > > > > > > 0x100013a9b9b object missing on disk; some files may be lost > > > > > > > > (/nextcloud/custom_apps/phonetrack) > > > > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 0 > > mds.0.cache.dir(0x100013a2659) > > > > > > > > _fetched missing object for [dir 0x100013a2659 > > > > > > > > /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 > > ap=1+0+0 > > > > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 > authpin=1 > > > > > > > > 0x55d4dcc41500] > > > > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 log_channel(cluster) log > > > > [ERR] : > > > > > > dir > > > > > > > > 0x100013a2659 object missing on disk; some files may be lost > > > > > > > > (/nextcloud/custom_apps/richdocuments) > > > > > > > > > 2019-11-06 20:20:22.360 7f0bd9090700 1 mds.k8s-node-01 > Updating > > MDS > > > > map > > > > > > to > > > > > > > > version 32015 from mon.1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Karsten > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for any hints > > > > > > > > > > > > > - Karsten > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

diesel home delivery
by powerbaba48＠gmail.com 12 Nov '19

12 Nov '19

diesel home delivery bangalore: Get Diesel fuel to your generators and machines with no restrictions on minimum order and save money by avoiding fuel theft. Home delivery of fuel in Bangalore is simple, call 8884144444 or place order online. https://powerbaba.com

1 0

french language training in bangalore
by speakingindiabangalore＠gmail.com 12 Nov '19

12 Nov '19

BONJOUR! Speakeng India Provides Best French Language Classes in Bangalore,With Our Certified Experts & Get 100% Assistance. http://www.speakengindia.com/french/

1 0

wedding makeup
by Glossandglassblr＠Gmail.Com 12 Nov '19

12 Nov '19

The GlossnGlass are the best bridal makeup artist in bangalore go beyond just creating immaculate makeup looks. We also train and produce the finest makeup artists through our professional makeup artist courses.With a team of expert trainers from the makeup industry, our makeup academy offers various intensive professional and personal grooming courses. https://www.glossnglass.com/

1 0

Bangalore b schools
by abbssm22＠gmail.com 12 Nov '19

12 Nov '19

<a href=https://www.abbssm.edu.in>Bangalore b schools</a>:best pgdm college in bangalore: Welcome to your future. We are the difference that makes you special. This is not just an institution, it's a springboard, and it's a catalyst. At ABBS, you will challenge yourself to learn, develop and r4e-engineer yourself to meet the demands of the country and the world. You are the future, and it is our responsibility to nurture the future.The MBA/PGDM at ABBS is specifically designed to prepare graduates in the emerging markets around the globe. The course is a transformative journey-offering unparalleled opportunity along with access to the best global management knowledge, corporate internships and placements from the finest companies in the market.this is one of the top 10 pgdm colleges in bangalore.

1 0

Re: mds crash loop
by Karsten Nielsen 12 Nov '19

12 Nov '19

-----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Mon 11-11-2019 15:09 Subject: Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Mon, Nov 11, 2019 at 5:09 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > I started a job that moved some files around in the cephfs cluster that > resulted in the mds to go back into the crash loop. > > Logs are here: > > http://s3.foo-bar.dk/mds-dumps/mds.log-20191111 > > > > Any help would be appriciated. > > > > looks like snaptable is corrupted. > > nautilus version (14.2.2) of ‘cephfs-data-scan scan_links’ can fix > snaptable. hopefully it will fix your issue. > > you don't need to upgrade whole cluster. Just install nautilus in a > temp machine or compile ceph from source. I did run the command that you suggested, it did not unfortunately fix the problem. http://s3.foo-bar.dk/mds-dumps/mds.log-20191112 > > > > > - Karsten > > > > -----Original message----- > > From: Yan, Zheng <ukernel(a)gmail.com> > > Sent: Thu 07-11-2019 14:20 > > Subject: Re: [ceph-users] Re: mds crash loop > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > CC: ceph-users(a)ceph.io; > > > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > > > That is awesome. > > > > > > > > Now I just need to figure out where the lost+found files needs to go. > > > > And what happened to the missing objects for the dirs. > > > > > > > > > > lost+found files are likely files that were deleted. you can keep the > > > lost+found dir for a while, then delete the 'lost+found' directory. > > > > > > for 'missing object' dirs, mv all of them to a temp directory, such as > > > /mnt/cephfs/missing_obj_dirs. > > > Then run command 'ceph daemon mds.x scrub_patch /missing_obj_dirs > > > force recursive repair'. wait a minute, the rm -rf > > > /mnt/cephfs/missing_obj_dirs > > > > > > > Any tool that is able to do that ? > > > > > > > > Thanks > > > > - Karsten > > > > > > > > -----Original message----- > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > Sent: Thu 07-11-2019 09:22 > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > CC: ceph-users(a)ceph.io; > > > > > I have tracked down the root cause. See > > > https://tracker.ceph.com/issues/42675 > > > > > > > > > > Regards > > > > > Yan, Zheng > > > > > > > > > > On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen <karsten(a)foo-bar.dk> > wrote: > > > > > > > > > > > > -----Original message----- > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > Sent: Thu 07-11-2019 07:21 > > > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen <karsten(a)foo-bar.dk> > > > wrote: > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > Sent: Wed 06-11-2019 14:16 > > > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen > <karsten(a)foo-bar.dk> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > > > Sent: Wed 06-11-2019 08:15 > > > > > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen > > > <karsten(a)foo-bar.dk> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > Last week I upgraded my ceph cluster from luminus to mimic > > > 13.2.6 > > > > > > > > > > > > It was running fine for a while but yesterday my mds went > > > into a > > > > > crash > > > > > > > > > loop. > > > > > > > > > > > > > > > > > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of > which > > > is > > > > > > > running > > > > > > > > > the > > > > > > > > > > > same crash loop. > > > > > > > > > > > > I am running ceph based on > > > https://hub.docker.com/r/ceph/daemon > > > > > > > version > > > > > > > > > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv > store. > > > > > > > > > > > > > > > > > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > > > > > > > > > > > > > > > > > Yan, Zheng > > > > > > > > > > > > > > > > > > > > Yes I have set that and had to move to pastebin.com as debian > > > > > apperently > > > > > > > only > > > > > > > > > supports 150k > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://pastebin.com/Gv7c5h54 > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looks like on-disk root inode is corrupted. have you > encountered any > > > > > > > > > unusually things during the upgrade? > > > > > > > > > > > > > > > > > > please run 'rados -p <cephfs metadata pool> stat > 1.00000000.inode' , > > > > > > > > > check if the object is modified before or after the 'luminous -> > > > > > > > > > 13.2.6' upgrade. > > > > > > > > > To fix the corrupted object. Run 'cephfs-data-scan init > > > > > > > > > --force-init'. Then restart mds. After mds become active, run > 'ceph > > > > > > > > > daemon mds.x scrub_path / force repair' > > > > > > > > > > > > > > > > > > > > > > > > > I followed the steps I got the mds started but now a lot of files > are > > > in > > > > > > > lost+found 24283 and I have these errors in the mds log > > > > > > > > > > > > > > > 'cephfs-data-scan init --force-init' does not move files into > > > > > > > lost+found. have you ever run other 'cephfs-data-scan foo' command > or > > > > > > > 'cephfs-journal-tool foo' command? > > > > > > > > > > > > I have had a similar problem with the cluster before where I went > through > > > the > > > > > cycle of: > > > > > > https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ -> Using an > > > > > alternate metadata pool for recovery > > > > > > > > > > > > I did run the cephfs-journal-tool journal reset command, mostly > because > > > > > cephfs is not that utilized so I thought it was safe to do as after the > > > upgrade > > > > > the cluster has not been used much, so data lose would be minimal - > > > apparently > > > > > I was wrong. > > > > > > > > > > > > > > > > > > > > > 2019-11-06 20:20:18.215 7f0bd9090700 1 mds.0.32011 cluster > recovered. > > > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 0 > mds.0.cache.dir(0x100013acfcb) > > > > > > > _fetched missing object for [dir 0x100013acfcb > > > > > /nextcloud/custom_apps/carnet/ > > > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100] > > > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 log_channel(cluster) log > > > [ERR] : > > > > > dir > > > > > > > 0x100013acfcb object missing on disk; some files may be lost > > > > > > > (/nextcloud/custom_apps/carnet) > > > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 0 > mds.0.cache.dir(0x100013a3156) > > > > > > > _fetched missing object for [dir 0x100013a3156 > > > /nextcloud/custom_apps/mail/ > > > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000] > > > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 log_channel(cluster) log > > > [ERR] : > > > > > dir > > > > > > > 0x100013a3156 object missing on disk; some files may be lost > > > > > > > (/nextcloud/custom_apps/mail) > > > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 0 > mds.0.cache.dir(0x100013abb3c) > > > > > > > _fetched missing object for [dir 0x100013abb3c > > > > > > > /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > > > 0x55d4dcc40700] > > > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 log_channel(cluster) log > > > [ERR] : > > > > > dir > > > > > > > 0x100013abb3c object missing on disk; some files may be lost > > > > > > > (/nextcloud/custom_apps/passwords) > > > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 0 > mds.0.cache.dir(0x100013a9b9b) > > > > > > > _fetched missing object for [dir 0x100013a9b9b > > > > > > > /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > > > 0x55d4dcc40e00] > > > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 log_channel(cluster) log > > > [ERR] : > > > > > dir > > > > > > > 0x100013a9b9b object missing on disk; some files may be lost > > > > > > > (/nextcloud/custom_apps/phonetrack) > > > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 0 > mds.0.cache.dir(0x100013a2659) > > > > > > > _fetched missing object for [dir 0x100013a2659 > > > > > > > /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 > ap=1+0+0 > > > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > > > 0x55d4dcc41500] > > > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 log_channel(cluster) log > > > [ERR] : > > > > > dir > > > > > > > 0x100013a2659 object missing on disk; some files may be lost > > > > > > > (/nextcloud/custom_apps/richdocuments) > > > > > > > > 2019-11-06 20:20:22.360 7f0bd9090700 1 mds.k8s-node-01 Updating > MDS > > > map > > > > > to > > > > > > > version 32015 from mon.1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Karsten > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for any hints > > > > > > > > > > > > - Karsten > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > >

1 0

Top wedding decorators in bangalore
by naadiamirzablr＠gmail.com 12 Nov '19

12 Nov '19

NaadiaMirza &CO was born on the day she realised that creativity was a gift given to her family and business acumen inherited by her father. Thus the company and each project always has a part of him in it . Naadia claims she was lucky to inherit passion and zest which is most important when putting together a big day. The idea is to create spaces that are personal and uniquely yet magical.Today, the team of designers work with local artisans and creative talent to celebrate your special day. <a href="https://www.naadiamirza.com/best-wedding-planner-in-bangalore/">Top wedding decorators in bangalore</a>

1 0

law schools in bangalore
by abbslaw123＠gmail.com 12 Nov '19

12 Nov '19

law courses in bangalore:The ABBS School of Law has started from the academic year 2018 – 19 under the aegis of Samagra Sikshana Samithi Trust. The Trust started the institution with a cherished ambition to impart legal education which is a very important step towards spreading legal awareness and empowering the helpless in the society. The College is affiliated to Karnataka State Law University, Hubballi and recognized by Bar Council of India, New Delhi. The college has a Governing Council duly constituted by Samagra Sikshana Samithi Trust. All the members of the Governing Council are accomplished academicians, legal luminaries, management experts and persons of eminence in public service. Governing Council meets regularly two times a year to review academic and administrative work. https://abbslaw.edu.in/

1 0

OSD's addrvec, not getting msgr v2 address, PGs stuck unknown or peering
by Wesley Dillingham 11 Nov '19

11 Nov '19

Running 14.2.4 (but same issue observed on 14.2.2) we have a problem with, thankfully a testing cluster, where all pgs are failing to peer and are stuck in peering or unknown stale etc states. My working theory is that this is because the OSDs dont seem to be utilizing msgr v2 as "ceph osd find osd.NN" only lists the v1 in the addrvec. This is in contrast to our working 14.2.4 clusters where both v1 and v2 are listed. Our monitors via `ceph mon dump` show each mon running on v1 and v2 on the default ports (3300/6789) and I able to reach each of those ports on all the mons from a few test OSD nodes. OSD logs are filled with heartbeat_check: no reply from <IP> <OSD.XY> ever on either front or back I have attempted to modify the ceph.conf mon_host on the OSDs to use either the standard comma separate ip list and the new bracketed format and then restarting OSD daemons on a number of OSDs but it doesnt seem to impact the addrvec. My desire is to get the OSDs working on V2 and see if they are able to begin peering. How can I force the addrvec to update? Thanks. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com

1 0

Re: mds crash loop
by Karsten Nielsen 11 Nov '19

11 Nov '19

I started a job that moved some files around in the cephfs cluster that resulted in the mds to go back into the crash loop. Logs are here: http://s3.foo-bar.dk/mds-dumps/mds.log-20191111 Any help would be appriciated. - Karsten -----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Thu 07-11-2019 14:20 Subject: Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > That is awesome. > > > > Now I just need to figure out where the lost+found files needs to go. > > And what happened to the missing objects for the dirs. > > > > lost+found files are likely files that were deleted. you can keep the > lost+found dir for a while, then delete the 'lost+found' directory. > > for 'missing object' dirs, mv all of them to a temp directory, such as > /mnt/cephfs/missing_obj_dirs. > Then run command 'ceph daemon mds.x scrub_patch /missing_obj_dirs > force recursive repair'. wait a minute, the rm -rf > /mnt/cephfs/missing_obj_dirs > > > Any tool that is able to do that ? > > > > Thanks > > - Karsten > > > > -----Original message----- > > From: Yan, Zheng <ukernel(a)gmail.com> > > Sent: Thu 07-11-2019 09:22 > > Subject: Re: [ceph-users] Re: mds crash loop > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > CC: ceph-users(a)ceph.io; > > > I have tracked down the root cause. See > https://tracker.ceph.com/issues/42675 > > > > > > Regards > > > Yan, Zheng > > > > > > On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > > > -----Original message----- > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > Sent: Thu 07-11-2019 07:21 > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > CC: ceph-users(a)ceph.io; > > > > > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen <karsten(a)foo-bar.dk> > wrote: > > > > > > > > > > > > -----Original message----- > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > Sent: Wed 06-11-2019 14:16 > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen <karsten(a)foo-bar.dk> > > > wrote: > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > Sent: Wed 06-11-2019 08:15 > > > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen > <karsten(a)foo-bar.dk> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Last week I upgraded my ceph cluster from luminus to mimic > 13.2.6 > > > > > > > > > > It was running fine for a while but yesterday my mds went > into a > > > crash > > > > > > > loop. > > > > > > > > > > > > > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of which > is > > > > > running > > > > > > > the > > > > > > > > > same crash loop. > > > > > > > > > > I am running ceph based on > https://hub.docker.com/r/ceph/daemon > > > > > version > > > > > > > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > > > > > > > > > > > > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > > > > > > > > > > > > > Yan, Zheng > > > > > > > > > > > > > > > > Yes I have set that and had to move to pastebin.com as debian > > > apperently > > > > > only > > > > > > > supports 150k > > > > > > > > > > > > > > > > > > > > > > > > https://pastebin.com/Gv7c5h54 > > > > > > > > > > > > > > > > > > > > > > Looks like on-disk root inode is corrupted. have you encountered any > > > > > > > unusually things during the upgrade? > > > > > > > > > > > > > > please run 'rados -p <cephfs metadata pool> stat 1.00000000.inode' , > > > > > > > check if the object is modified before or after the 'luminous -> > > > > > > > 13.2.6' upgrade. > > > > > > > To fix the corrupted object. Run 'cephfs-data-scan init > > > > > > > --force-init'. Then restart mds. After mds become active, run 'ceph > > > > > > > daemon mds.x scrub_path / force repair' > > > > > > > > > > > > > > > > > > > I followed the steps I got the mds started but now a lot of files are > in > > > > > lost+found 24283 and I have these errors in the mds log > > > > > > > > > > > 'cephfs-data-scan init --force-init' does not move files into > > > > > lost+found. have you ever run other 'cephfs-data-scan foo' command or > > > > > 'cephfs-journal-tool foo' command? > > > > > > > > I have had a similar problem with the cluster before where I went through > the > > > cycle of: > > > > https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ -> Using an > > > alternate metadata pool for recovery > > > > > > > > I did run the cephfs-journal-tool journal reset command, mostly because > > > cephfs is not that utilized so I thought it was safe to do as after the > upgrade > > > the cluster has not been used much, so data lose would be minimal - > apparently > > > I was wrong. > > > > > > > > > > > > > > > 2019-11-06 20:20:18.215 7f0bd9090700 1 mds.0.32011 cluster recovered. > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 0 mds.0.cache.dir(0x100013acfcb) > > > > > _fetched missing object for [dir 0x100013acfcb > > > /nextcloud/custom_apps/carnet/ > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100] > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013acfcb object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/carnet) > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a3156) > > > > > _fetched missing object for [dir 0x100013a3156 > /nextcloud/custom_apps/mail/ > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000] > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a3156 object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/mail) > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 0 mds.0.cache.dir(0x100013abb3c) > > > > > _fetched missing object for [dir 0x100013abb3c > > > > > /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc40700] > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013abb3c object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/passwords) > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a9b9b) > > > > > _fetched missing object for [dir 0x100013a9b9b > > > > > /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc40e00] > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a9b9b object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/phonetrack) > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a2659) > > > > > _fetched missing object for [dir 0x100013a2659 > > > > > /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc41500] > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a2659 object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/richdocuments) > > > > > > 2019-11-06 20:20:22.360 7f0bd9090700 1 mds.k8s-node-01 Updating MDS > map > > > to > > > > > version 32015 from mon.1 > > > > > > > > > > > > > > > > > > > > > > > > > > > - Karsten > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for any hints > > > > > > > > > > - Karsten > > > > > > > > > > _______________________________________________ > > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

Where rocksdb on my OSD's?
by Andrey Groshev 11 Nov '19

11 Nov '19

Hello, Some time ago I deployed a ceph cluster. It works great. Today I collect some statistics and found that the BlueFs utility is not working. # ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-8 inferring bluefs devices from bluestore path slot 1 /var/lib/ceph/osd/ceph-8/block -> /dev/dm-5 unable to open /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) _lock flock failed on /var/lib/ceph/osd/ceph-8/block 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) open failed to lock /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable As far as I understand, blocks.db and blocks.wal are missing. I don’t know how it turned out. # ls -l /var/lib/ceph/osd/ceph-8 total 28 lrwxrwxrwx 1 ceph ceph 93 Nov 9 15:56 block -> /dev/ceph-55b8a53d-1740-402a-b6f4-09d4befdd564/osd-block-c5488db7-621a-490a-88a0-904c12e8b8ed -rw------- 1 ceph ceph 37 Nov 9 15:56 ceph_fsid -rw------- 1 ceph ceph 37 Nov 9 15:56 fsid -rw------- 1 ceph ceph 55 Nov 9 15:56 keyring -rw------- 1 ceph ceph 6 Nov 9 15:56 ready -rw-r--r-- 1 ceph ceph 3 Nov 9 15:56 require_osd_release -rw------- 1 ceph ceph 10 Nov 9 15:56 type -rw------- 1 ceph ceph 2 Nov 9 15:56 whoami Did as standard: .... ceph-deploy osd create test-host1:/dev/sde ceph-deploy osd create test-host2:/dev/sde .... What now to do and whether it is necessary to repair it at all? Assembled on luminous, yesterday updated to Nautilus.

2 4

Past_interval start interval mismatch (last_clean_epoch reported)
by Huseyin Cotuk 10 Nov '19

10 Nov '19

Hi all, I also hit the bug #24866 in my test environment. According to the logs, the last_clean_epoch in the specified OSD/PG is 17703, but the interval starts with 17895. So the OSD fails to start. There are some other OSDs in the same status. 2019-10-14 18:22:51.908 7f0a275f1700 -1 osd.21 pg_epoch: 18432 pg[18.51( v 18388'4 lc 18386'3 (0'0,18388'4] local-lis/les=18430/18431 n=1 ec=295/295 lis/c 18430/17702 les/c/f 18431/17703/0 18428/18430/18421) [11,21]/[11,21,20] r=1 lpr=18431 pi=[17895,18430)/3 crt=18388'4 lcod 0'0 unknown m=1 mbc={}] 18.51 past_intervals [17895,18430) start interval does not contain the required bound [17703,18430) start The cause is pg 18.51 went clean in 17703 but 17895 is reported to the monitor. I am using the last stable version of Mimic (13.2.6). Any idea how to fix it? Is there any way to bypass this check or fix the reported epoch #? Thanks in advance. Best regards, Huseyin Cotuk hcotuk(a)gmail.com

3 2

Re: Wrong %USED and MAX AVAIL stats for pool
by Yordan Yordanov (Innologica) 09 Nov '19

09 Nov '19

The documentation states: https://docs.ceph.com/docs/mimic/rados/operations/monitoring/ The POOLS section of the output provides a list of pools and the notional usage of each pool. The output from this section DOES NOT reflect replicas, clones or snapshots. For example, if you store an object with 1MB of data, the notional usage will be 1MB, but the actual usage may be 2MB or more depending on the number of replicas, clones and snapshots. However in our case we are clearly seeing the USAGE field multiplying the total object sizes to the number of replicas. [root@blackmirror ~]# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 80 TiB 34 TiB 46 TiB 46 TiB 58.10 TOTAL 80 TiB 34 TiB 46 TiB 46 TiB 58.10 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL one 2 15 TiB 4.05M 46 TiB 68.32 7.2 TiB bench 5 250 MiB 67 250 MiB 0 22 TiB [root@blackmirror ~]# rbd du -p one NAME PROVISIONED USED ... <TOTAL> 20 TiB 15 TiB This is causing several apps (including ceph dashboard) to display inaccurate percentages, because they calculate the total pool capacity as USED + MAX AVAIL, which in this case yields 53.2TB, which is way off. 7.2TB is about 13% of that, so we receive alarms and this is bugging us for quite some time now.

4 4

Re: mds crash loop
by Karsten Nielsen 08 Nov '19

08 Nov '19

-----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Thu 07-11-2019 14:20 Subject: Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > That is awesome. > > > > Now I just need to figure out where the lost+found files needs to go. > > And what happened to the missing objects for the dirs. > > > > lost+found files are likely files that were deleted. you can keep the > lost+found dir for a while, then delete the 'lost+found' directory. > > for 'missing object' dirs, mv all of them to a temp directory, such as > /mnt/cephfs/missing_obj_dirs. > Then run command 'ceph daemon mds.x scrub_patch /missing_obj_dirs > force recursive repair'. wait a minute, the rm -rf > /mnt/cephfs/missing_obj_dirs I got this output from that command. 2019-11-08 14:27:53.110 7f0bda092700 1 mds.k8s-node-01 asok_command: scrub_path (starting...) 2019-11-08 14:27:53.110 7f0bda092700 1 mds.k8s-node-01 asok_command: scrub_path (complete) 2019-11-08 14:27:55.794 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013a3156(/missing_obj_dirs/mail), rewriting it 2019-11-08 14:27:55.794 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013a3156 (/missing_obj_dirs/mail) 2019-11-08 14:27:55.794 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013a3156 [...2,head] /missing_obj_dirs/mail/ auth v77565 dirtyparent f(v0 m2019-11-07 10:10:35.293698 12=3+9) n(v52 rc2019-11-08 14:08:21.585140 b21180195 2561=1820+741) (inest lock dirty) (ifile excl) (iversion lock) caps={1793815=pAsXsFs/-@5},l=1793815 | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=1 scrubqueue=0 dirty=1 waiter=0 authpin=0 0x55d4dc882300]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013a3156:[<0x100026d3cff/mail v77565>,<0x1/missing_obj_dirs v3025317>]//","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2019-11-07 10:10:35.293698 12=3+9)","ondisk_value.rstat":"n(v0 rc2019-11-07 1 0:10:38.061742 b21180195 2561=1820+741)","memory_value.dirrstat":"f(v0 m2019-11-07 10:10:35.293698 12=3+9)","memory_value.rstat":"n(v52 rc2019-11-08 14:08:21.585140 b21180195 2561=1820+741)","error_str":""},"return_code":-61} 2019-11-08 14:27:55.794 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013950d6(/missing_obj_dirs/empty_templates), rewriting it 2019-11-08 14:27:55.794 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013950d6 (/missing_obj_dirs/empty_templates) 2019-11-08 14:27:55.794 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013950d6 [...2,head] /missing_obj_dirs/empty_templates/ auth v77569 ap=2+0 dirtyparent f(v0 m2019-09-17 19:13:12.226251 3=3+0) n(v0 rc2019-11-08 14:09:01.450030 b29712 4=3+1) (inest lock) (ifile sync->lock) (iversion lock) caps={1793815=pAsXs/pAsXsFs/-@4,1804200=pAsLsXs/pAsLsXsFs/AsLsXsFsx@31} | request=1 lock=0 dirfrag=0 caps=1 dirtyparent=1 scrubqueue=0 dirty=1 waiter=1 authpin=1 0x55d4ee870000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013950d6:[<0x100026d3cff/empty_templates v77569>,<0x1/missing_obj_dirs v3025317>]//","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value. dirrstat":"f(v0 m2019-09-17 19:13:12.226251 3=3+0)","memory_value.rstat":"n(v0 rc2019-11-08 14:09:01.450030 b29712 4=3+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2019-11-08 14:27:56.114 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013abb3c(/missing_obj_dirs/passwords), rewriting it 2019-11-08 14:27:56.114 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013abb3c (/missing_obj_dirs/passwords) 2019-11-08 14:27:56.114 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013abb3c [...2,head] /missing_obj_dirs/passwords/ auth v77567 dirtyparent f(v0 m2019-11-07 10:03:19.194734 8=1+7) n(v20 rc2019-11-08 14:08:30.565341 b5715351 413=349+64) (inest lock) (ifile excl) (iversion lock) caps={1793815=pAsXsFs/-@6},l=1793815 | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=1 scrubqueue=0 dirty=1 waiter=0 authpin=0 0x55d4dc880000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013abb3c:[<0x100026d3cff/passwords v77567>,<0x1/missing_obj_dirs v3025512>]//","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2019-11-07 10:03:19.194734 8=1+7)","ondisk_value.rstat":"n(v0 rc2019-11-07 10: 03:24.178813 b5715351 413=349+64)","memory_value.dirrstat":"f(v0 m2019-11-07 10:03:19.194734 8=1+7)","memory_value.rstat":"n(v20 rc2019-11-08 14:08:30.565341 b5715351 413=349+64)","error_str":""},"return_code":-61} 2019-11-08 14:27:56.254 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013a2659(/missing_obj_dirs/richdocuments), rewriting it 2019-11-08 14:27:56.254 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013a2659 (/missing_obj_dirs/richdocuments) 2019-11-08 14:27:56.254 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013a2659 [...2,head] /missing_obj_dirs/richdocuments/ auth v77561 dirtyparent f(v0 m2019-11-07 14:54:47.223440 19=9+10) n(v30 rc2019-11-08 14:08:03.876745 b3205323 261=241+20) (inest lock) (ifile excl) (iversion lock) caps={1793815=pAsXsFs/-@3},l=1793815 | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=1 scrubqueue=0 dirty=1 waiter=0 authpin=0 0x55d4dc87e300]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013a2659:[<0x100026d3cff/richdocuments v77561>,<0x1/missing_obj_dirs v3025512>]//","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2019-11-07 14:54:47.223440 19=9+10)","ondisk_value.rstat":"n(v0 rc20 19-11-07 14:54:47.223440 b3205323 261=241+20)","memory_value.dirrstat":"f(v0 m2019-11-07 14:54:47.223440 19=9+10)","memory_value.rstat":"n(v30 rc2019-11-08 14:08:03.876745 b3205323 261=241+20)","error_str":""},"return_code":-61} 2019-11-08 14:27:56.254 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100013acfcb(/missing_obj_dirs/carnet), rewriting it 2019-11-08 14:27:56.254 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100013acfcb (/missing_obj_dirs/carnet) 2019-11-08 14:27:56.254 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100013acfcb [...2,head] /missing_obj_dirs/carnet/ auth v77563 ap=2+0 dirtyparent f(v0 m2019-10-21 11:03:29.113564 20=11+9) n(v31 rc2019-11-08 14:08:09.480870 b12005493 1999=1656+343) (inest lock) (ifile sync->lock) (iversion lock) caps={1793815=pAsXs/pAsXsFs/-@4,1794668=pAsXs/pAsXsFs/-@108394} | request=1 lock=0 dirfrag=0 caps=1 dirtyparent=1 scrubqueue=0 dirty=1 waiter=1 authpin=1 0x55d4dc889800]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100013acfcb:[<0x100026d3cff/carnet v77563>,<0x1/missing_obj_dirs v3025512>]//","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirrstat":"f (v0 m2019-10-21 11:03:29.113564 20=11+9)","memory_value.rstat":"n(v31 rc2019-11-08 14:08:09.480870 b12005493 1999=1656+343)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2019-11-08 14:27:56.258 7f0bd2dfa700 0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x100026d3cff(/missing_obj_dirs), rewriting it 2019-11-08 14:27:56.258 7f0bd2dfa700 0 log_channel(cluster) log [INF] : Scrub repaired inode 0x100026d3cff (/missing_obj_dirs) 2019-11-08 14:27:56.258 7f0bd2dfa700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100026d3cff [...2,head] /missing_obj_dirs/ auth v3025512 pv3025514 ap=1+1 dirtyparent f(v0 m2019-11-08 14:09:01.450030 5=0+5) n(v0 rc2019-11-08 14:09:01.450030 b42106362 5236=4066+1170) (inest lock w=1) (ifile excl) (iversion lock w=1) caps={1793815=pAsLsXsFsx/AsLsXsFsx@7},l=1793815 | request=0 lock=2 dirfrag=1 caps=1 dirtyparent=1 scrubqueue=1 dirty=1 authpin=1 0x55d4ede56700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//","memoryvalue":"(6)0x100026d3cff:[<0x1/missing_obj_dirs v3025512>]//","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2019-11-08 14:09:01.450030 5=0+5)","ondisk_value.rstat":"n(v0 rc2019-11-08 14:09:01.450030 b42106362 5236=4 066+1170)","memory_value.dirrstat":"f(v0 m2019-11-08 14:09:01.450030 5=0+5)","memory_value.rstat":"n(v0 rc2019-11-08 14:09:01.450030 b42106362 5236=4066+1170)","error_str":""},"return_code":-2} 2019-11-08 14:27:56.258 7f0bd2dfa700 0 log_channel(cluster) log [INF] : scrub complete I am not sure what to read out of that. Other than data seems to be missing. > > > Any tool that is able to do that ? > > > > Thanks > > - Karsten > > > > -----Original message----- > > From: Yan, Zheng <ukernel(a)gmail.com> > > Sent: Thu 07-11-2019 09:22 > > Subject: Re: [ceph-users] Re: mds crash loop > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > CC: ceph-users(a)ceph.io; > > > I have tracked down the root cause. See > https://tracker.ceph.com/issues/42675 > > > > > > Regards > > > Yan, Zheng > > > > > > On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > > > -----Original message----- > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > Sent: Thu 07-11-2019 07:21 > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > CC: ceph-users(a)ceph.io; > > > > > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen <karsten(a)foo-bar.dk> > wrote: > > > > > > > > > > > > -----Original message----- > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > Sent: Wed 06-11-2019 14:16 > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen <karsten(a)foo-bar.dk> > > > wrote: > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > Sent: Wed 06-11-2019 08:15 > > > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen > <karsten(a)foo-bar.dk> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Last week I upgraded my ceph cluster from luminus to mimic > 13.2.6 > > > > > > > > > > It was running fine for a while but yesterday my mds went > into a > > > crash > > > > > > > loop. > > > > > > > > > > > > > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of which > is > > > > > running > > > > > > > the > > > > > > > > > same crash loop. > > > > > > > > > > I am running ceph based on > https://hub.docker.com/r/ceph/daemon > > > > > version > > > > > > > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > > > > > > > > > > > > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > > > > > > > > > > > > > Yan, Zheng > > > > > > > > > > > > > > > > Yes I have set that and had to move to pastebin.com as debian > > > apperently > > > > > only > > > > > > > supports 150k > > > > > > > > > > > > > > > > > > > > > > > > https://pastebin.com/Gv7c5h54 > > > > > > > > > > > > > > > > > > > > > > Looks like on-disk root inode is corrupted. have you encountered any > > > > > > > unusually things during the upgrade? > > > > > > > > > > > > > > please run 'rados -p <cephfs metadata pool> stat 1.00000000.inode' , > > > > > > > check if the object is modified before or after the 'luminous -> > > > > > > > 13.2.6' upgrade. > > > > > > > To fix the corrupted object. Run 'cephfs-data-scan init > > > > > > > --force-init'. Then restart mds. After mds become active, run 'ceph > > > > > > > daemon mds.x scrub_path / force repair' > > > > > > > > > > > > > > > > > > > I followed the steps I got the mds started but now a lot of files are > in > > > > > lost+found 24283 and I have these errors in the mds log > > > > > > > > > > > 'cephfs-data-scan init --force-init' does not move files into > > > > > lost+found. have you ever run other 'cephfs-data-scan foo' command or > > > > > 'cephfs-journal-tool foo' command? > > > > > > > > I have had a similar problem with the cluster before where I went through > the > > > cycle of: > > > > https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ -> Using an > > > alternate metadata pool for recovery > > > > > > > > I did run the cephfs-journal-tool journal reset command, mostly because > > > cephfs is not that utilized so I thought it was safe to do as after the > upgrade > > > the cluster has not been used much, so data lose would be minimal - > apparently > > > I was wrong. > > > > > > > > > > > > > > > 2019-11-06 20:20:18.215 7f0bd9090700 1 mds.0.32011 cluster recovered. > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 0 mds.0.cache.dir(0x100013acfcb) > > > > > _fetched missing object for [dir 0x100013acfcb > > > /nextcloud/custom_apps/carnet/ > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100] > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013acfcb object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/carnet) > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a3156) > > > > > _fetched missing object for [dir 0x100013a3156 > /nextcloud/custom_apps/mail/ > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000] > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a3156 object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/mail) > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 0 mds.0.cache.dir(0x100013abb3c) > > > > > _fetched missing object for [dir 0x100013abb3c > > > > > /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc40700] > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013abb3c object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/passwords) > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a9b9b) > > > > > _fetched missing object for [dir 0x100013a9b9b > > > > > /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc40e00] > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a9b9b object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/phonetrack) > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a2659) > > > > > _fetched missing object for [dir 0x100013a2659 > > > > > /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > > > 0x55d4dcc41500] > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 log_channel(cluster) log > [ERR] : > > > dir > > > > > 0x100013a2659 object missing on disk; some files may be lost > > > > > (/nextcloud/custom_apps/richdocuments) > > > > > > 2019-11-06 20:20:22.360 7f0bd9090700 1 mds.k8s-node-01 Updating MDS > map > > > to > > > > > version 32015 from mon.1 > > > > > > > > > > > > > > > > > > > > > > > > > > > - Karsten > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for any hints > > > > > > > > > > - Karsten > > > > > > > > > > _______________________________________________ > > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Nautilus beast rgw 2 minute delay on startup???
by Marc Roos 08 Nov '19

08 Nov '19

Why is it taking for rgw 2 minutes to be available? Is there some internet lookup being done that is timing out when there is no default gw? [@ ~]# /usr/bin/radosgw -d -f --cluster ceph --name client.rgw2 --keyring=/etc/ceph/ceph.client.rgw2.keyring 2019-11-08 12:32:04.930 7fc9793fb780 0 framework: beast 2019-11-08 12:32:04.930 7fc9793fb780 0 framework conf key: ssl_port, val: 7480 2019-11-08 12:32:04.930 7fc9793fb780 0 framework conf key: ssl_private_key, val: /etc/pki/tls/private/rgw.xxxxxxx.xxx.key 2019-11-08 12:32:04.930 7fc9793fb780 0 framework conf key: ssl_certificate, val: /etc/pki/tls/certs/rgw.xxxxxxx.xxx.crt 2019-11-08 12:32:04.930 7fc9793fb780 0 ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable), process radosgw, pid 177335 2019-11-08 12:34:12.541 7fc9793fb780 0 starting handler: beast 2019-11-08 12:34:12.556 7fc9793fb780 1 mgrc service_daemon_register rgw.rgw2 metadata {arch=x86_64,ceph_release=nautilus,ceph_version=ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable),ceph_version_short=14.2.4,cpu=Intel,distro=centos,distro_descri ption=CentOS Linux 7 (Core),distro_version=7,frontend_config#0=beast ssl_port=7480 ssl_private_key=/etc/pki/tls/private/rgw.xxxxxxx.xxx.key ssl_certificate=/etc/pki/tls/certs/rgw.xxxxxxx.xxx.crt,frontend_type#0=b east,hostname=,kernel_description=#1 SMP Fri Sep 13 22:55:44 UTC 2019,kernel_version=3.10.0-1062.1.1.el7.x86_64,mem_swap_kb=10485756,mem_ total_kb=98796656,num_handles=1,os=Linux,pid=177335,zone_id=91613b5d-6b9 b-4848-9546-2479a112cbff,zone_name=default,zonegroup_id=1a766e64-8d85-48 c4-9fae-96391d524a49,zonegroup_name=default}

1 0

Proper way to replace an OSD with a shared SSD for db/wal
by Hayashida, Mami 08 Nov '19

08 Nov '19

We are running the Mimic version of Ceph (13.2.6) and I would like to know a proper way of replacing a defective OSD disk that has its DB and WAL on a separate SSD drive which is shared with 9 other OSDs. More specifically, the failing disk for osd.327 is on /dev/sdai and its wal/db are on /dev/sdc, which is partitioned into 10 LVs, holding wal/db for osd.320-329. When I deployed it, I used pv/vg/lvcreate commands to make VG named ssd1, LV named db320, db321 and so on. Then I used the ceph-deploy command from an admin node (`ceph-deploy osd create --block-db=ssd1/db327 --data=dev/sdai <node>`). My main question is what to do about the separate wal/db data as this page ( https://docs.ceph.com/docs/mimic/rados/operations/add-or-rm-osds/) does not seem to address the issue. 1) Do I need to erase the wal/db data on the ssd1/db327 Logical Volume? If so, how should I do that? 2) Assuming 1) is taken care of (and the "old" OSD is destroyed and the "bad" hard drive has been physically replaced with a new one), does this command look correct? `ceph-volume lvm create --osd-id 327 --bluestore --data /dev/sdai --block.db ssd1/db327` *Mami Hayashida* *Research Computing Associate* Univ. of Kentucky ITS Research Computing Infrastructure

2 1

best schools in sarjapur road
by foundationschool school 08 Nov '19

08 Nov '19

The Foundation schools were established in 2009. The Gunjur branch has become one of the top CBSE schools in varthur, Gunjur, Whitefield and best school in Sarjapur Road areas in a short period of time. The schools follow a methodology that helps children through hands on activities and experiential learning. Children at The Foundation School are seen not as consumers who need external entertainment, but as producers who interact with various learning material with their own hands; construct knowledge and gain immense satisfaction from the process. In keeping with this methodology, The Foundation School follows the Montessori Philosophy at the pre-primary level, and experiential learning from grade 1 onwards. All of these make The Foundation School, one of the foremost leading and progressive schools in Varthur, Gunjur, Schools in Whitefield and also schools near sarjapur Road areas of Bangalore. https://www.foundationschoolindia.com/

1 0

mgr daemons becoming unresponsive
by Oliver Freyermuth 07 Nov '19

07 Nov '19

Dear Cephers, this is a 14.2.4 cluster with device health metrics enabled - since about a day, all mgr daemons go "silent" on me after a few hours, i.e. "ceph -s" shows: cluster: id: 269cf2b2-7e7c-4ceb-bd1b-a33d915ceee9 health: HEALTH_WARN no active mgr 1/3 mons down, quorum mon001,mon002 services: mon: 3 daemons, quorum mon001,mon002 (age 57m), out of quorum: mon003 mgr: no daemons active (since 56m) ... (the third mon has a planned outage and will come back in a few days) Checking the logs of the mgr daemons, I find some "reset" messages at the time when it goes "silent", first for the first mgr: 2019-11-01 21:34:40.286 7f2df6a6b700 0 log_channel(cluster) log [DBG] : pgmap v1798: 1585 pgs: 1585 active+clean; 1.1 TiB data, 2.3 TiB used, 136 TiB / 138 TiB avail 2019-11-01 21:34:41.458 7f2e0d59b700 0 client.0 ms_handle_reset on v2:10.160.16.1:6800/401248 2019-11-01 21:34:42.287 7f2df6a6b700 0 log_channel(cluster) log [DBG] : pgmap v1799: 1585 pgs: 1585 active+clean; 1.1 TiB data, 2.3 TiB used, 136 TiB / 138 TiB avail and a bit later, on the standby mgr: 2019-11-01 22:18:14.892 7f7bcc8ae700 0 log_channel(cluster) log [DBG] : pgmap v1798: 1585 pgs: 166 active+clean+snaptrim, 858 active+clean+snaptrim_wait, 561 active+clean; 1.1 TiB data, 2.3 TiB used, 136 TiB / 138 TiB avail 2019-11-01 22:18:16.022 7f7be9e72700 0 client.0 ms_handle_reset on v2:10.160.16.2:6800/352196 2019-11-01 22:18:16.893 7f7bcc8ae700 0 log_channel(cluster) log [DBG] : pgmap v1799: 1585 pgs: 166 active+clean+snaptrim, 858 active+clean+snaptrim_wait, 561 active+clean; 1.1 TiB data, 2.3 TiB used, 136 TiB / 138 TiB avail Interestingly, the dashboard still works, but presents outdated information, and for example zero I/O going on. I believe this started to happen mainly after the third mon went into the known downtime, but I am not fully sure if this was the trigger, since the cluster is still growing. It may also have been the addition of 24 more OSDs. I also find other messages in the mgr logs which seem problematic, but I am not sure they are related: ------------------------------ 2019-11-01 21:17:09.849 7f2df4266700 0 mgr[devicehealth] Error reading OMAP: [errno 22] Failed to operate read op for oid Traceback (most recent call last): File "/usr/share/ceph/mgr/devicehealth/module.py", line 396, in put_device_metrics ioctx.operate_read_op(op, devid) File "rados.pyx", line 516, in rados.requires.wrapper.validate_func (/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.4/rpm/el7/BUIL D/ceph-14.2.4/build/src/pybind/rados/pyrex/rados.c:4721) File "rados.pyx", line 3474, in rados.Ioctx.operate_read_op (/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.4/rpm/el7/BUILD/ceph-14.2.4/build/src/pybind/rados/pyrex/rados.c:36554) InvalidArgumentError: [errno 22] Failed to operate read op for oid ------------------------------ or: ------------------------------ 2019-11-01 21:33:53.977 7f7bd38bc700 0 mgr[devicehealth] Fail to parse JSON result from daemon osd.51 () 2019-11-01 21:33:53.978 7f7bd38bc700 0 mgr[devicehealth] Fail to parse JSON result from daemon osd.52 () 2019-11-01 21:33:53.979 7f7bd38bc700 0 mgr[devicehealth] Fail to parse JSON result from daemon osd.53 () ------------------------------ The reason why I am cautious about the health metrics is that I observed a crash when trying to query them: ------------------------------ 2019-11-01 20:21:23.661 7fa46314a700 0 log_channel(audit) log [DBG] : from='client.174136 -' entity='client.admin' cmd=[{"prefix": "device get-health-metrics", "devid": "osd.11", "target": ["mgr", ""]}]: dispatch 2019-11-01 20:21:23.661 7fa46394b700 0 mgr[devicehealth] handle_command 2019-11-01 20:21:23.663 7fa46394b700 -1 *** Caught signal (Segmentation fault) ** in thread 7fa46394b700 thread_name:mgr-fin ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) 1: (()+0xf5f0) [0x7fa488cee5f0] 2: (PyEval_EvalFrameEx()+0x1a9) [0x7fa48aeb50f9] 3: (PyEval_EvalFrameEx()+0x67bd) [0x7fa48aebb70d] 4: (PyEval_EvalFrameEx()+0x67bd) [0x7fa48aebb70d] 5: (PyEval_EvalFrameEx()+0x67bd) [0x7fa48aebb70d] 6: (PyEval_EvalCodeEx()+0x7ed) [0x7fa48aebe08d] 7: (()+0x709c8) [0x7fa48ae479c8] 8: (PyObject_Call()+0x43) [0x7fa48ae22ab3] 9: (()+0x5aaa5) [0x7fa48ae31aa5] 10: (PyObject_Call()+0x43) [0x7fa48ae22ab3] 11: (()+0x4bb95) [0x7fa48ae22b95] 12: (PyObject_CallMethod()+0xbb) [0x7fa48ae22ecb] 13: (ActivePyModule::handle_command(std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&, ceph::buffer::v14_2_0::list const&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*)+0x20e) [0x55c3c1fefc5e] 14: (()+0x16c23d) [0x55c3c204023d] 15: (FunctionContext::finish(int)+0x2c) [0x55c3c2001eac] 16: (Context::complete(int)+0x9) [0x55c3c1ffe659] 17: (Finisher::finisher_thread_entry()+0x156) [0x7fa48b439cc6] 18: (()+0x7e65) [0x7fa488ce6e65] 19: (clone()+0x6d) [0x7fa48799488d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. ------------------------------ I have issued: ceph device monitoring off for now and will keep waiting to see if mgrs go silent again. If there are any better ideas or this issue is known, let me know. Cheers, Oliver

8 25

RGW compression not compressing
by Bryan Stillwell 07 Nov '19

07 Nov '19

Today I tried enabling RGW compression on a Nautilus 14.2.4 test cluster and found it wasn't doing any compression at all. I figure I must have missed something in the docs, but I haven't been able to find out what that is and could use some help. This is the command I used to enable zlib-based compression: # radosgw-admin zone placement modify --rgw-zone=default --placement-id=default-placement --compression=zlib I then restarted the radosgw process to activate the change (there's only 1 RGW in this test cluster): # systemctl restart ceph-radosgw(a)radosgw.$(hostname -s) I verified it was enabled correctly with: # radosgw-admin zone get | jq -r '.placement_pools' [ { "key": "default-placement", "val": { "index_pool": "default.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "default.rgw.buckets.data", "compression_type": "zlib" } }, "data_extra_pool": "default.rgw.buckets.non-ec", "index_type": 0 } } ] Before starting the test I had nothing in the default.rgw.buckets.data pool: # ceph df | grep default.rgw.buckets.data default.rgw.buckets.data 16 0 B 0 0 B 0 230 TiB I then tried uploading a 1GiB file consisting of all 0s from /dev/zero with both S3 (boto) and Swift (swiftclient) and each time they used 1GiB of data on the cluster: # ceph df -f json | jq -r '.' | grep -A9 default.rgw.buckets.data "name": "default.rgw.buckets.data", "id": 16, "stats": { "stored": 1073741824, "objects": 256, "kb_used": 1048576, "bytes_used": 1073741824, "percent_used": 1.4138463484414387e-06, "max_avail": 253148744646656 } The same thing was reported by bucket stats: # radosgw-admin bucket stats --bucket=bs-test | jq -r '.usage' { "rgw.main": { "size": 1073741824, "size_actual": 1073741824, "size_utilized": 1073741824, "size_kb": 1048576, "size_kb_actual": 1048576, "size_kb_utilized": 1048576, "num_objects": 1 } } What am I missing? Thanks, Bryan

2 4

Device Health Metrics on EL 7
by Oliver Freyermuth 07 Nov '19

07 Nov '19

Dear Cephers, I went through some of the OSD logs of our 14.2.4 nodes and found this: ---------------------------------- Nov 01 01:22:25 sudo[1087697]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -a --json /dev/sds Nov 01 01:22:51 sudo[1087729]: pam_unix(sudo:auth): conversation failed Nov 01 01:22:51 sudo[1087729]: pam_unix(sudo:auth): auth could not identify password for [ceph] Nov 01 01:22:51 sudo[1087729]: pam_succeed_if(sudo:auth): requirement "uid >= 1000" not met by user "ceph" Nov 01 01:22:53 sudo[1087729]: ceph : command not allowed ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=nvme lvm smart-log-add --json /dev/sds ---------------------------------- It seems with device health metrics, the OSDs try to run smartctl with "sudo", which expectedly fails, since the Ceph user (as system user) has a uid smaller than 1000. Also, it's of course not in /etc/sudoers. Does somebody have a working setup with device health metrics which could be shared (and documented, or made part of future packaging ;-) ) ? Cheers, Oliver

2 3

Balancer is active, but not balancing
by Thomas Schneider 07 Nov '19

07 Nov '19

Hi, I activated balancer in order to balance data distribution: root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } However, the data stored on 1.6TB HDD in specific pool "hdb_backup" is not balanced; the range starts with osd.265 size: 1.6 usage: 52.83 reweight: 1.00000 and ends with osd.145 size: 1.6 usage: 80.19 reweight: 1.00000 The affected drives are located on 4 nodes. The result is that not all available disk space is available for usage. I have attached pastebin <https://pastebin.com/dNyEwNR0> with - ceph osd df sorted by usage - ceph osd df tree My cluster has multiple crush roots respresenting different disks. In addition I have defined multiple pools, one pool for each disk type: hdd, ssd, nvme. I need to get the distribution better based on pools. Please advise how to start balancer to correct data distribution. THX

1 0

Re: mds crash loop
by Karsten Nielsen 07 Nov '19

07 Nov '19

That is awesome. Now I just need to figure out where the lost+found files needs to go. And what happened to the missing objects for the dirs. Any tool that is able to do that ? Thanks - Karsten -----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Thu 07-11-2019 09:22 Subject: Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > I have tracked down the root cause. See https://tracker.ceph.com/issues/42675 > > Regards > Yan, Zheng > > On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > -----Original message----- > > From: Yan, Zheng <ukernel(a)gmail.com> > > Sent: Thu 07-11-2019 07:21 > > Subject: Re: [ceph-users] Re: mds crash loop > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > CC: ceph-users(a)ceph.io; > > > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > > > -----Original message----- > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > Sent: Wed 06-11-2019 14:16 > > > > Subject: Re: [ceph-users] mds crash loop > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > CC: ceph-users(a)ceph.io; > > > > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen <karsten(a)foo-bar.dk> > wrote: > > > > > > > > > > > > -----Original message----- > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > Sent: Wed 06-11-2019 08:15 > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen <karsten(a)foo-bar.dk> > > > wrote: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > Last week I upgraded my ceph cluster from luminus to mimic 13.2.6 > > > > > > > > It was running fine for a while but yesterday my mds went into a > crash > > > > > loop. > > > > > > > > > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of which is > > > running > > > > > the > > > > > > > same crash loop. > > > > > > > > I am running ceph based on https://hub.docker.com/r/ceph/daemon > > > version > > > > > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > > > > > > > > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > > > > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > > > > > > > > > Yan, Zheng > > > > > > > > > > > > Yes I have set that and had to move to pastebin.com as debian > apperently > > > only > > > > > supports 150k > > > > > > > > > > > > > > > > > > https://pastebin.com/Gv7c5h54 > > > > > > > > > > > > > > > > Looks like on-disk root inode is corrupted. have you encountered any > > > > > unusually things during the upgrade? > > > > > > > > > > please run 'rados -p <cephfs metadata pool> stat 1.00000000.inode' , > > > > > check if the object is modified before or after the 'luminous -> > > > > > 13.2.6' upgrade. > > > > > To fix the corrupted object. Run 'cephfs-data-scan init > > > > > --force-init'. Then restart mds. After mds become active, run 'ceph > > > > > daemon mds.x scrub_path / force repair' > > > > > > > > > > > > > I followed the steps I got the mds started but now a lot of files are in > > > lost+found 24283 and I have these errors in the mds log > > > > > > > 'cephfs-data-scan init --force-init' does not move files into > > > lost+found. have you ever run other 'cephfs-data-scan foo' command or > > > 'cephfs-journal-tool foo' command? > > > > I have had a similar problem with the cluster before where I went through the > cycle of: > > https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ -> Using an > alternate metadata pool for recovery > > > > I did run the cephfs-journal-tool journal reset command, mostly because > cephfs is not that utilized so I thought it was safe to do as after the upgrade > the cluster has not been used much, so data lose would be minimal - apparently > I was wrong. > > > > > > > > > 2019-11-06 20:20:18.215 7f0bd9090700 1 mds.0.32011 cluster recovered. > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 0 mds.0.cache.dir(0x100013acfcb) > > > _fetched missing object for [dir 0x100013acfcb > /nextcloud/custom_apps/carnet/ > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100] > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : > dir > > > 0x100013acfcb object missing on disk; some files may be lost > > > (/nextcloud/custom_apps/carnet) > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a3156) > > > _fetched missing object for [dir 0x100013a3156 /nextcloud/custom_apps/mail/ > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000] > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : > dir > > > 0x100013a3156 object missing on disk; some files may be lost > > > (/nextcloud/custom_apps/mail) > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 0 mds.0.cache.dir(0x100013abb3c) > > > _fetched missing object for [dir 0x100013abb3c > > > /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > 0x55d4dcc40700] > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : > dir > > > 0x100013abb3c object missing on disk; some files may be lost > > > (/nextcloud/custom_apps/passwords) > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a9b9b) > > > _fetched missing object for [dir 0x100013a9b9b > > > /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > 0x55d4dcc40e00] > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : > dir > > > 0x100013a9b9b object missing on disk; some files may be lost > > > (/nextcloud/custom_apps/phonetrack) > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a2659) > > > _fetched missing object for [dir 0x100013a2659 > > > /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > > > 0x55d4dcc41500] > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : > dir > > > 0x100013a2659 object missing on disk; some files may be lost > > > (/nextcloud/custom_apps/richdocuments) > > > > 2019-11-06 20:20:22.360 7f0bd9090700 1 mds.k8s-node-01 Updating MDS map > to > > > version 32015 from mon.1 > > > > > > > > > > > > > > > > > > > - Karsten > > > > > > > > > > > > > > > > > > > > > Thanks for any hints > > > > > > > > - Karsten > > > > > > > > _______________________________________________ > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

Disabling keep alive with rgw beast
by Dan van der Ster 07 Nov '19

07 Nov '19

Hi all, We are experiencing some sporadic 502's on our traefik/rgw setup. It's similar to the issue described here: https://github.com/containous/traefik/issues/3237 The solution seems to be to disable keep-alive in the traefik and rgw configurations. We found the option for civetweb (enable_keep_alive=no) but didn't find the equivalent for beast. Any ideas? Thanks! Dan

1 0

Re: mds crash loop
by Karsten Nielsen 07 Nov '19

07 Nov '19

-----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Thu 07-11-2019 07:21 Subject: Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > -----Original message----- > > From: Yan, Zheng <ukernel(a)gmail.com> > > Sent: Wed 06-11-2019 14:16 > > Subject: Re: [ceph-users] mds crash loop > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > CC: ceph-users(a)ceph.io; > > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > > > -----Original message----- > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > Sent: Wed 06-11-2019 08:15 > > > > Subject: Re: [ceph-users] mds crash loop > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > CC: ceph-users(a)ceph.io; > > > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen <karsten(a)foo-bar.dk> > wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > Last week I upgraded my ceph cluster from luminus to mimic 13.2.6 > > > > > > It was running fine for a while but yesterday my mds went into a crash > > > loop. > > > > > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of which is > running > > > the > > > > > same crash loop. > > > > > > I am running ceph based on https://hub.docker.com/r/ceph/daemon > version > > > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > > > > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > > > > > Yan, Zheng > > > > > > > > Yes I have set that and had to move to pastebin.com as debian apperently > only > > > supports 150k > > > > > > > > > > > > https://pastebin.com/Gv7c5h54 > > > > > > > > > > Looks like on-disk root inode is corrupted. have you encountered any > > > unusually things during the upgrade? > > > > > > please run 'rados -p <cephfs metadata pool> stat 1.00000000.inode' , > > > check if the object is modified before or after the 'luminous -> > > > 13.2.6' upgrade. > > > To fix the corrupted object. Run 'cephfs-data-scan init > > > --force-init'. Then restart mds. After mds become active, run 'ceph > > > daemon mds.x scrub_path / force repair' > > > > > > > I followed the steps I got the mds started but now a lot of files are in > lost+found 24283 and I have these errors in the mds log > > > 'cephfs-data-scan init --force-init' does not move files into > lost+found. have you ever run other 'cephfs-data-scan foo' command or > 'cephfs-journal-tool foo' command? I have had a similar problem with the cluster before where I went through the cycle of: https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ -> Using an alternate metadata pool for recovery I did run the cephfs-journal-tool journal reset command, mostly because cephfs is not that utilized so I thought it was safe to do as after the upgrade the cluster has not been used much, so data lose would be minimal - apparently I was wrong. > > > 2019-11-06 20:20:18.215 7f0bd9090700 1 mds.0.32011 cluster recovered. > > 2019-11-06 20:20:19.019 7f0bd2dfa700 0 mds.0.cache.dir(0x100013acfcb) > _fetched missing object for [dir 0x100013acfcb /nextcloud/custom_apps/carnet/ > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100] > > 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : dir > 0x100013acfcb object missing on disk; some files may be lost > (/nextcloud/custom_apps/carnet) > > 2019-11-06 20:20:19.275 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a3156) > _fetched missing object for [dir 0x100013a3156 /nextcloud/custom_apps/mail/ > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000] > > 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : dir > 0x100013a3156 object missing on disk; some files may be lost > (/nextcloud/custom_apps/mail) > > 2019-11-06 20:20:19.371 7f0bd2dfa700 0 mds.0.cache.dir(0x100013abb3c) > _fetched missing object for [dir 0x100013abb3c > /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > 0x55d4dcc40700] > > 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : dir > 0x100013abb3c object missing on disk; some files may be lost > (/nextcloud/custom_apps/passwords) > > 2019-11-06 20:20:19.383 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a9b9b) > _fetched missing object for [dir 0x100013a9b9b > /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > 0x55d4dcc40e00] > > 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : dir > 0x100013a9b9b object missing on disk; some files may be lost > (/nextcloud/custom_apps/phonetrack) > > 2019-11-06 20:20:19.431 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a2659) > _fetched missing object for [dir 0x100013a2659 > /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 ap=1+0+0 > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 > 0x55d4dcc41500] > > 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : dir > 0x100013a2659 object missing on disk; some files may be lost > (/nextcloud/custom_apps/richdocuments) > > 2019-11-06 20:20:22.360 7f0bd9090700 1 mds.k8s-node-01 Updating MDS map to > version 32015 from mon.1 > > > > > > > > > > > - Karsten > > > > > > > > > > > > > > > Thanks for any hints > > > > > > - Karsten > > > > > > _______________________________________________ > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

Re: mds crash loop
by Karsten Nielsen 07 Nov '19

07 Nov '19

-----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Wed 06-11-2019 14:16 Subject: Re: [ceph-users] mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > -----Original message----- > > From: Yan, Zheng <ukernel(a)gmail.com> > > Sent: Wed 06-11-2019 08:15 > > Subject: Re: [ceph-users] mds crash loop > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > CC: ceph-users(a)ceph.io; > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > > > Hi, > > > > > > > > Last week I upgraded my ceph cluster from luminus to mimic 13.2.6 > > > > It was running fine for a while but yesterday my mds went into a crash > loop. > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of which is running > the > > > same crash loop. > > > > I am running ceph based on https://hub.docker.com/r/ceph/daemon version > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > Yan, Zheng > > > > Yes I have set that and had to move to pastebin.com as debian apperently only > supports 150k > > > > > > https://pastebin.com/Gv7c5h54 > > > > Looks like on-disk root inode is corrupted. have you encountered any > unusually things during the upgrade? > > please run 'rados -p <cephfs metadata pool> stat 1.00000000.inode' , > check if the object is modified before or after the 'luminous -> > 13.2.6' upgrade. > To fix the corrupted object. Run 'cephfs-data-scan init > --force-init'. Then restart mds. After mds become active, run 'ceph > daemon mds.x scrub_path / force repair' > I followed the steps I got the mds started but now a lot of files are in lost+found 24283 and I have these errors in the mds log 2019-11-06 20:20:18.215 7f0bd9090700 1 mds.0.32011 cluster recovered. 2019-11-06 20:20:19.019 7f0bd2dfa700 0 mds.0.cache.dir(0x100013acfcb) _fetched missing object for [dir 0x100013acfcb /nextcloud/custom_apps/carnet/ [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100] 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : dir 0x100013acfcb object missing on disk; some files may be lost (/nextcloud/custom_apps/carnet) 2019-11-06 20:20:19.275 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a3156) _fetched missing object for [dir 0x100013a3156 /nextcloud/custom_apps/mail/ [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000] 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : dir 0x100013a3156 object missing on disk; some files may be lost (/nextcloud/custom_apps/mail) 2019-11-06 20:20:19.371 7f0bd2dfa700 0 mds.0.cache.dir(0x100013abb3c) _fetched missing object for [dir 0x100013abb3c /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40700] 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : dir 0x100013abb3c object missing on disk; some files may be lost (/nextcloud/custom_apps/passwords) 2019-11-06 20:20:19.383 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a9b9b) _fetched missing object for [dir 0x100013a9b9b /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40e00] 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : dir 0x100013a9b9b object missing on disk; some files may be lost (/nextcloud/custom_apps/phonetrack) 2019-11-06 20:20:19.431 7f0bd2dfa700 0 mds.0.cache.dir(0x100013a2659) _fetched missing object for [dir 0x100013a2659 /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc41500] 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 log_channel(cluster) log [ERR] : dir 0x100013a2659 object missing on disk; some files may be lost (/nextcloud/custom_apps/richdocuments) 2019-11-06 20:20:22.360 7f0bd9090700 1 mds.k8s-node-01 Updating MDS map to version 32015 from mon.1 > > > - Karsten > > > > > > > > > Thanks for any hints > > > > - Karsten > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > >

2 1