September 2019 - ceph-users

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

ceph osd set-require-min-compat-client jewel failure
by 潘东元 14 May '20

14 May '20

hi,every one, my ceph version 12.2.12，I want to set require min compat client luminous,I use command #ceph osd set-require-min-compat-client luminous but ceph report:Error EPERM: cannot set require_min_compat_client to luminous: 4 connected client(s) look like jewel (missing 0xa00000000200000); add --yes-i-really-mean-it to do it anyway [root@node-1 ~]# ceph features { "mon": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 3 } }, "osd": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 15 } }, "client": { "group": { "features": "0x40106b84a842a52", "release": "jewel", "num": 4 }, "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 168 } } } so,I run command: [root@node-1 gyt]# ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it set require_min_compat_client to luminous but now,I want to set require min compat client jewel,I use command： [root@node-1 gyt]# ceph osd set-require-min-compat-client jewel Error EPERM: osdmap current utilizes features that require luminous; cannot set require_min_compat_client below that to jewel what‘s the way we are set luminous chang to jewel？

3 2

osd_pg_create causing slow requests in Nautilus
by Bryan Stillwell 03 Apr '20

03 Apr '20

We've run into a problem on our test cluster this afternoon which is running Nautilus (14.2.2). It seems that any time PGs move on the cluster (from marking an OSD down, setting the primary-affinity to 0, or by using the balancer), a large number of the OSDs in the cluster peg the CPU cores they're running on for a while which causes slow requests. From what I can tell it appears to be related to slow peering caused by osd_pg_create() taking a long time. This was seen on quite a few OSDs while waiting for peering to complete: # ceph daemon osd.3 ops { "ops": [ { "description": "osd_pg_create(e179061 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:34:46.556413", "age": 318.25234538000001, "duration": 318.25241895300002, "type_data": { "flag_point": "started", "events": [ { "time": "2019-08-27 14:34:46.556413", "event": "initiated" }, { "time": "2019-08-27 14:34:46.556413", "event": "header_read" }, { "time": "2019-08-27 14:34:46.556299", "event": "throttled" }, { "time": "2019-08-27 14:34:46.556456", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456901", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456903", "event": "wait for new map" }, { "time": "2019-08-27 14:40:01.292346", "event": "started" } ] } }, ...snip... { "description": "osd_pg_create(e179066 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:35:09.908567", "age": 294.900191001, "duration": 294.90068416899999, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-08-27 14:35:09.908567", "event": "initiated" }, { "time": "2019-08-27 14:35:09.908567", "event": "header_read" }, { "time": "2019-08-27 14:35:09.908520", "event": "throttled" }, { "time": "2019-08-27 14:35:09.908617", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456921", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456923", "event": "wait for new map" } ] } } ], "num_ops": 6 } That "wait for new map" message made us think something was getting hung up on the monitors, so we restarted them all without any luck. I'll keep investigating, but so far my google searches aren't pulling anything up so I wanted to see if anyone else is running into this? Thanks, Bryan

6 16

official ceph.com buster builds?
by Chad W Seys 27 Feb '20

27 Feb '20

Hi all, Am I missing the ceph buster build built by ceph.com ? http://download.ceph.com/debian-nautilus/dists/ Should I be using the Croit supplied builds? Thanks! Chad.

2 2

MDS failing under load with large cache sizes
by Janek Bevendorff 07 Jan '20

07 Jan '20

Hi, I am trying to copy the contents of our storage server into a CephFS, but am experiencing stability issues with my MDSs. The CephFS sits on top of an erasure-coded pool with 5 MONs, 5 MDSs and a max_mds setting of two. My Ceph cluster version is Nautilus, the client is Mimic and uses the kernel module to mount the FS. The index of filenames to copy is about 23GB and I am using 16 parallel rsync processes over a 10G link to copy the files over to Ceph. This works perfectly for a while, but then the MDSs start reporting oversized caches (between 20 and 50GB, sometimes more) and an inode count between 1 and 4 million. Particularly the Inode count seems quite high to me. Each rsync job has 25k files to work with, so if all 16 processes open all their files at the same time, I should not exceed 400k. Even if I double this number to account for the client's page cache, I should get nowhere near that number of inodes (a sync flush takes about 1 second). Then after a few hours, my MDSs start failing with messages like this: -21> 2019-07-22 14:00:05.877 7f67eacec700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -20> 2019-07-22 14:00:05.877 7f67eacec700 0 mds.beacon.XXX Skipping beacon heartbeat to monitors (last acked 24.0042s ago); MDS internal heartbeat is not healthy! The standby nodes try to take over, but take forever to become active and will fail as well eventually. During my research, I found this related topic: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015959.html, but I tried everything in there from increasing to lowering my cache size, the number of segments etc. I also played around with the number of active MDSs and two appears to work the best, whereas one cannot keep up with the load and three seems to be the worst of all choices. Do you have any ideas how I can improve the stability of my MDS damons to handle the load properly? single 10G link is a toy and we could query the cluster with a lot more requests per second, but it's already yielding to 16 rsync processes. Thanks

4 33

Re: Authentication failure at radosgw for presigned urls
by Biswajeet Patra 25 Oct '19

25 Oct '19

++ceph-users On Fri, Sep 20, 2019 at 4:48 PM Biswajeet Patra < biswajeet.patra(a)flipkart.com> wrote: > Hi, > We recently faced an issue with radosgw authentication of presigned urls. > The presigned url generated by client to download object fails at radosgw > with http error 403 i.e SignatureDoesNotMatch. > > The radosgw computes the signature (*v2 signature in this case*) using > the s3 specification > https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html. > In this process, a string_to_sign is created by concatenating selected > elements from requests which is then authenticated using hmac to create the > final signature. If this signature matches with the client signature, the > authentication is successful or else it fails with SignatureDoesNotMatch > error. As part of the StringToSign, the CanonicalizedAmzHeaders should only > include headers that start with "x-amz-" and ignore other headers. But in > the radosgw code, there are other meta prefixes that are checked against > the http request headers and if matched are included in the > CanonicalizedAmzHeaders to compute the final signature. For e.g, if a > request header contains "HTTP_X_ACCOUNT" its selected by radosgw to include > in amz_headers but the same will be ignored by AWS SDK as it does not start > with "x-amz-". This will result in different signature computed by client > and radosgw. > > Code Snippet: rgw_common.cc > struct str_len meta_prefixes[] = { STR_LEN_ENTRY("HTTP_X_AMZ"), > STR_LEN_ENTRY("HTTP_X_GOOG"), > STR_LEN_ENTRY("HTTP_X_DHO"), > STR_LEN_ENTRY("HTTP_X_RGW"), > STR_LEN_ENTRY("HTTP_X_OBJECT"), > STR_LEN_ENTRY("HTTP_X_CONTAINER"), > STR_LEN_ENTRY("HTTP_X_ACCOUNT"), > {NULL, 0} }; > > The method init_meta_info() which matches the above prefixes is called > from RGWREST::preprocess() which is invoked for all s3 requests. It will be > helpful to know as to why these prefixes that are not specified by AWS S3 > comes in the path of s3 authentication. Was it added for swift use-cases > only? If so, then why its included in rgw_common.cc? > As a proposed fix, we can remove the highlighted meta prefixes that are > not specified by aws from the s3 authentication path signature calculation. > Let me know if you have any queries or solutions. > > Regards, > Biswajeet > -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 1

Nautilus: BlueFS spillover
by Eugen Block 07 Oct '19

07 Oct '19

Hi, I'm following the discussion for a tracker issue [1] about spillover warnings that affects our upgraded Nautilus cluster. Just to clarify, would a resize of the rocksDB volume (and expanding with 'ceph-bluestore-tool bluefs-bdev-expand...') resolve that or do we have to recreate every OSD? Regards, Eugen [1] https://tracker.ceph.com/issues/38745

4 8

one read/write, many read only
by khaled atteya 01 Oct '19

01 Oct '19

Hi, Is it possible to do this scenario : If one open a file first , he will get read/write permissions and other will get read-only permission if they open the file after the first one. Thanks

2 1

Doubt about ceph-iscsi and Vmware
by Gesiel Galvão Bernardes 01 Oct '19

01 Oct '19

Hi, I'm testing Ceph with Vmware, using Ceph-iscsi gateway. I reading documentation* and have doubts some points: - If I understanded, in general terms, for each VMFS datastore in VMware will match the an RBD image. (consequently in an RBD image I will possible have many VMWare disks). Its correct? - In documentation is this: "gwcli requires a pool with the name rbd, so it can store metadata like the iSCSI configuration". In part 4 of "Configuration", have: "Add a RBD image with the name disk_1 in the pool rbd". In this part, the use of "rbd" pool is a example and I could use any pool for storage of image, or the pool should be "rbd"? Resuming: gwcli require "rbd" pool for metadata and I could use any pool for image, or i will use just "rbd pool" for storage image and metadata? - How much memory ceph-iscsi use? Which is a good number of RAM? Regards Gesiel * https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/

4 5

moving EC pool from HDD to SSD without downtime
by Frank Schilder 01 Oct '19

01 Oct '19

I need to move a 6+2 EC pool from HDDs to SSDs while storage must remain accessible. All SSDs and HDDs are within the same failure domains. The crush rule in question is rule sr-rbd-data-one { id 5 type erasure min_size 3 max_size 8 step set_chooseleaf_tries 50 step set_choose_tries 1000 step take ServerRoom class hdd step chooseleaf indep 0 type host step emit } and I would be inclined just to change the entry "step take ServerRoom class hdd" to "step take ServerRoom class ssd" and wait for the dust to settle. However, this will almost certainly lead to all PGs being undersized and inaccessible as all objects are in the wrong place. I noticed that this is not an issue with PGs created by replicated rules as they can contain more OSDs than the replication factor while objects are moved. The same does not apply to EC rules. I suspect this is due to the setting "max_size 8", which does not allow for more than 6+2=8 OSDs being a member of a PG. What is the correct way to do what I need to do? Can I just set "max_size 16" and go? Will this work with EC rules? If not, what are my options? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

Ceph pool capacity question...
by Ilmir Mulyukov 01 Oct '19

01 Oct '19

Hello! We have a small proxmox farm with ceph consisting of three nodes. Each node has 6 disks each with a capacity of 4 TB. A only one pool has been created on these disks. Size 2/1. In theory, this pool should have a capacity: 32.74 TB But the ceph df command returns only: 22.4 TB (USED + MAX AVAIL)(16.7 + 5.7) How to explain this difference? *ceph version is:* 12.2.12-pve1 *ceph df command out:* POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED ala01vf01p01 7 N/A N/A 16.7TiB 74.53 5.70TiB 4411119 4.41M 2.62GiB 887MiB 33.4TiB *crush map:* host n01vf01 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily id -18 class nvme # do not change unnecessarily # weight 22.014 alg straw2 hash 0 # rjenkins1 item osd.0 weight 3.669 item osd.13 weight 3.669 item osd.14 weight 3.669 item osd.15 weight 3.669 item osd.16 weight 3.669 item osd.17 weight 3.669 } host n02vf01 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily id -19 class nvme # do not change unnecessarily # weight 22.014 alg straw2 hash 0 # rjenkins1 item osd.1 weight 3.669 item osd.8 weight 3.669 item osd.9 weight 3.669 item osd.10 weight 3.669 item osd.11 weight 3.669 item osd.12 weight 3.669 } host n04vf01 { id -34 # do not change unnecessarily id -35 class hdd # do not change unnecessarily id -36 class nvme # do not change unnecessarily # weight 22.014 alg straw2 hash 0 # rjenkins1 item osd.7 weight 3.669 item osd.27 weight 3.669 item osd.24 weight 3.669 item osd.25 weight 3.669 item osd.26 weight 3.669 item osd.28 weight 3.669 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily id -21 class nvme # do not change unnecessarily # weight 66.042 alg straw2 hash 0 # rjenkins1 item n01vf01 weight 22.014 item n02vf01 weight 22.014 item n04vf01 weight 22.014 } rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit }

2 2

Crush device class switchover
by Reed Dier 30 Sep '19

30 Sep '19

I currently have two roots in my crush map, one for HDD devices and one for SSD devices, and have had it that way since Jewel. I am currently on Nautilus, and have had my crush device classes for my OSD's set since Luminous. > ID CLASS WEIGHT TYPE NAME > -13 105.37599 root ssd > -11 105.37599 rack ssd.rack2 > -14 17.61099 host ceph00 > 24 ssd 1.76099 osd.24 > -1 398.92554 root default > -10 397.07343 rack default.rack2 > -70 44.45032 chassis ceph05 > -67 44.45032 host ceph05 > 74 hdd 1.85210 osd.74 I have crush rulesets that distribute based on the roots for each device class. > [ > { > "rule_id": 0, > "rule_name": "replicated_ruleset", > "ruleset": 0, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -1, > "item_name": "default" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "chassis" > }, > { > "op": "emit" > } > ] > }, > { > "rule_id": 1, > "rule_name": "ssd_ruleset", > "ruleset": 1, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -13, > "item_name": "ssd" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "host" > }, > { > "op": "emit" > } > ] > }, > { > "rule_id": 2, > "rule_name": "hybrid_ruleset", > "ruleset": 2, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -13, > "item_name": "ssd" > }, > { > "op": "chooseleaf_firstn", > "num": 1, > "type": "host" > }, > { > "op": "emit" > }, > { > "op": "take", > "item": -1, > "item_name": "default" > }, > { > "op": "chooseleaf_firstn", > "num": -1, > "type": "chassis" > }, > { > "op": "emit" > } > ] > } If I wanted to migrate to rulesets based on device class with minimal disruption, what are my options? In my mind the way this would work would be to 1. Set the norebalance flag 2. Rework my crush rulesets to use takes based on class rather than root. 3. Merge my ssd hosts from the ssd root to the default root 4. Let things rebalance? Would prefer minimal data movement, as that would be potentially disruptive, and I imagine provide minimal gain for me, but possibly better data distribution? Maybe there are better steps to take? Appreciate any help. Reed

2 1

PG is stuck in repmapped and degraded
by 展荣臻（信泰） 30 Sep '19

30 Sep '19

Hi,all we use openstack + ceph(hammer) in my production.There are 22 osds on a host and 11 osds share one ssd for osd journal. Unfortunately, one of the ssds does not work,so the 11 osds were down.The osd log shows as below: -1> 2019-09-19 11:35:52.681142 7fcab5354700 1 -- xxxxxxxxxxxx:6831/16460 --> xxxxxxxxxxxx:0/14304 -- osd_ping(ping_reply e6152 stamp 2019-09-19 11:35:52.679939) v2 -- ?+0 0x20af8400 con 0x20a4b340 0> 2019-09-19 11:35:52.682578 7fcabed3c700 -1 os/FileJournal.cc: In function 'void FileJournal::write_finish_thread_entry()' thread 7fcabed3c700 time 2019-09-19 11:35:52.640294 os/FileJournal.cc: 1426: FAILED assert(0 == "unexpected aio error") ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbc8b55] 2: (FileJournal::write_finish_thread_entry()+0x695) [0xa795c5] 3: (FileJournal::WriteFinisher::entry()+0xd) [0x91cecd] 4: (()+0x7dc5) [0x7fcacb81cdc5] 5: (clone()+0x6d) [0x7fcaca2fd1cd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this As it shows,osd was down at 09-19,But now the pg still is degraded and remapped,It seam stuck. ceph -s cluster 26cc714c-ed78-4d62-9435-db3e87509c5f health HEALTH_WARN 681 pgs degraded 681 pgs stuck degraded 797 pgs stuck unclean 681 pgs stuck undersized 681 pgs undersized recovery 132155/11239182 objects degraded (1.176%) recovery 22056/11239182 objects misplaced (0.196%) monmap e1: 3 mons at {ctrl01=xxx.xxx.xxx.xxx:6789/0,ctrl02=xxx.xxx.xxx.xxx:6789/0,ctrl03=xxx.xxx.xxx.xxx:6789/0} election epoch 122, quorum 0,1,2 ctrl01,ctrl02,ctrl03 osdmap e6590: 324 osds: 313 up, 313 in; 116 remapped pgs pgmap v40849600: 21504 pgs, 6 pools, 14048 GB data, 3658 kobjects 41661 GB used, 279 TB / 319 TB avail 132155/11239182 objects degraded (1.176%) 22056/11239182 objects misplaced (0.196%) 20707 active+clean 681 active+undersized+degraded 116 active+remapped client io 121 MB/s rd, 144 MB/s wr, 1029 op/s I query one of the pgs,the recovery_state is started [1] I also found pg have not third osd to mapped ,as shows below. [root@ctrl01 ~]# ceph pg map 4.75f osdmap e6590 pg 4.75f (4.75f) -> up [34,106] acting [34,106] crushmap at [2] How i should to get the cluster come back ok? can someone help me. very very thanks. [1] https://github.com/rongzhen-zhan/myfile/blob/master/pgquery [2] https://github.com/rongzhen-zhan/myfile/blob/master/crushmap

4 5

How to set read only mode to radosgw user?
by Charles Alva 29 Sep '19

29 Sep '19

Hi Cephalopods, I'm in the process of migrating radosgw Erasure Code pool from old cluster to Replica pool on new cluster. To avoid user write new object to old pool, I want to set the radosgw user privilege to read only. Could you guys please share how to limit radosgw user privilege to read only? I could not find any clear explanation and example in the Ceph radosgw-admin docs. Is it by changing the user's caps or op_mask? Or setting the civetweb option to only allow HTTP HEAD and GET methods? Kind regards, Charles Alva Sent from Gmail Mobile

1 0

RAM recommendation with large OSDs?
by Darrell Enns 27 Sep '19

27 Sep '19

The standard advice is "1GB RAM per 1TB of OSD". Does this actually still hold with large OSDs on bluestore? Can it be reasonably reduced with tuning? From the docs, it looks like bluestore should target the "osd_memory_target" value by default. This is a fixed value (4GB by default), which does not depend on OSD size. So shouldn't the advice really by "4GB per OSD", rather than "1GB per TB"? Would it also be reasonable to reduce osd_memory_target for further RAM savings? For example, suppose we have 90 12TB OSD drives: * "1GB per TB" rule: 1080GB RAM * "4GB per OSD" rule: 360GB RAM * "2GB per OSD" (osd_memory_target reduced to 2GB): 180GB RAM Those are some massively different RAM values. Perhaps the old advice was for filestore? Or there is something to consider beyond the bluestore memory target? What about when using very dense nodes (for example, 60 12TB OSDs on a single node)?

1 0

Raw use 10 times higher than data use
by Georg F 27 Sep '19

27 Sep '19

Hi all, I've recently moved a 1TiB pool (3TiB raw use) from hdd osds (7) to newly added nvme osds (14). The hdd osds should be almost empty by now as just small pools reside on them. The pools on the hdd osds in sum store about 25GiB, which should use about 75GiB with a pool size of 3. Wal and db are on separate devices. However the outputs of ceph df and ceph osd df tell a different story: # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 19 TiB 18 TiB 775 GiB 782 GiB 3.98 # ceph osd df | egrep "(ID|hdd)" ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 8 hdd 2.72392 1.00000 2.8 TiB 111 GiB 10 GiB 111 KiB 1024 MiB 2.7 TiB 3.85 0.60 65 up 6 hdd 2.17914 1.00000 2.3 TiB 112 GiB 11 GiB 83 KiB 1024 MiB 2.2 TiB 4.82 0.75 58 up 3 hdd 2.72392 1.00000 2.8 TiB 114 GiB 13 GiB 71 KiB 1024 MiB 2.7 TiB 3.94 0.62 76 up 5 hdd 2.72392 1.00000 2.8 TiB 109 GiB 7.6 GiB 83 KiB 1024 MiB 2.7 TiB 3.76 0.59 63 up 4 hdd 2.72392 1.00000 2.8 TiB 112 GiB 11 GiB 55 KiB 1024 MiB 2.7 TiB 3.87 0.60 59 up 7 hdd 2.72392 1.00000 2.8 TiB 114 GiB 13 GiB 8 KiB 1024 MiB 2.7 TiB 3.93 0.61 66 up 2 hdd 2.72392 1.00000 2.8 TiB 111 GiB 9.9 GiB 78 KiB 1024 MiB 2.7 TiB 3.84 0.60 69 up The sum of "DATA" is 75,5GiB which is what I am expecting to be used by the pools. How come the sum of "RAW USE" is 783GiB? More than 10x the size of the stored data. On my nvme osds the "RAW USE" to "DATA" overhead is <1%: ceph osd df|egrep "(ID|nvme)" ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 nvme 2.61989 1.00000 2.6 TiB 181 GiB 180 GiB 31 KiB 1.0 GiB 2.4 TiB 6.74 1.05 12 up 1 nvme 2.61989 1.00000 2.6 TiB 151 GiB 150 GiB 39 KiB 1024 MiB 2.5 TiB 5.62 0.88 10 up 13 nvme 2.61989 1.00000 2.6 TiB 239 GiB 238 GiB 55 KiB 1.0 GiB 2.4 TiB 8.89 1.39 16 up -- truncated -- I am running ceph version 14.2.3 (0f776cf838a1ae3130b2b73dc26be9c95c6ccc39) nautilus (stable) which was upgraded recently from 13.2.1. Any help is appreciated. Best regards, Georg

3 3

Re: Nfs-ganesha 2.6 upgrade to 2.7
by Daniel Gryniewicz 27 Sep '19

27 Sep '19

Ganesha itself has no dependencies on samba (and there aren't any on my system, when I build). These must be being pulled in by something else that Ganesha does use. Daniel On Thu, Sep 26, 2019 at 11:21 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > Is it really necessary to have these dependencies in nfs-ganesha 2.7 > > Dep-Install samba-client-libs-4.8.3-4.el7.x86_64 @CentOS7 > Dep-Install samba-common-4.8.3-4.el7.noarch @CentOS7 > Dep-Install samba-common-libs-4.8.3-4.el7.x86_64 @CentOS7 > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

HELP! Way too much space consumption with ceph-fuse using erasure code data pool under highly concurrent writing operations
by daihongbo＠yahoo.com 27 Sep '19

27 Sep '19

Hi, It's observerd up to 10 times space is consumed when concurrent 200 files iozone writing test , with erasure code profile (k=8,m=4) data pool, mounted with ceph fuse, but disk usage is normal if only has one writing task . Furthermore everything is normal using replicated data pool, no matter how many writing operations at the same time. Regards, Dai #iozone -s 100M -r 1M -i 0 -u 200 -l 200 -+n -w #df -h /data01 ceph-fuse on /data01 type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) # du -sh /data01 801M /data01 #ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 357 TiB 356 TiB 60 GiB 132 GiB 0.04 ssd 1.3 TiB 1.3 TiB 2.6 GiB 5.6 GiB 0.42 TOTAL 358 TiB 358 TiB 62 GiB 137 GiB 0.04 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL meta_data01 1 166 MiB 64 509 MiB 0.04 423 GiB data_data01 2 800 MiB 201 8.9 GiB 0 226 TiB #ceph osd pool get data_data01 erasure_code_profile erasure_code_profile: profile_data01 #ceph osd erasure-code-profile get profile_data01 crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=8 m=4 plugin=jerasure technique=reed_sol_van w=8

1 0

Check backend type
by 徐蕴 27 Sep '19

27 Sep '19

Hi, Is there a command to check whether an OSD is running filestore or bluestore? BR, Xu Yun

3 4

Slow Write Issues
by João Victor Rodrigues Soares 26 Sep '19

26 Sep '19

Hello, In my company, we currently have the following infrastructure: - Ceph Luminous - OpenStack Pike. We have a cluster of 3 osd nodes with the following configuration: - 1 x Xeon (R) D-2146NT CPU @ 2.30GHz - 128GB RAM - 128GB ROOT DISK - 12 x 10TB SATA ST10000NM0146 (OSD) - 1 x Intel Optane P4800X SSD DC 375GB (block.DB / block.wal) - Ubuntu 16.04 - 2 X 10Gb network interface configured with lacp The compute nodes have - 4 x 10Gb network interfaces with lacp. We also have 4 monitors with: - 4 x 10Gb lacp network interfaces. - The monitor nodes are approx. 90% cpu idle time with 32GB / 256GB available RAM For each OSD disk we have created a partition of 33GB to block.db and block.wal. We are recently facing a number of performance issues. Virtual machines created in OpenStack are experiencing slow writing issues (approx. 50MB / s). The OSD nodes monitoring incur an average of 20% cpu IOwait time and 70 cpu idle time. The memory consumption is around 30% consumption. We have no latency issues (9ms average) My question is if what is happening may have to do with the amount of disk dedicated to DB / WAL. In the CEPH documentation it says it is recommended that the block.db size is not smaller than 4% of block. In this case for each disk in my environment block.db could not be less than 400GB / OSD. Another question is if I set my disks to use block.db / block.wal on the mechanical disks themselves, if that could lead to a performance degradation. Att. João Victor Rodrigues Soares

4 5

Nfs-ganesha 2.6 upgrade to 2.7
by Marc Roos 26 Sep '19

26 Sep '19

Is it really necessary to have these dependencies in nfs-ganesha 2.7 Dep-Install samba-client-libs-4.8.3-4.el7.x86_64 @CentOS7 Dep-Install samba-common-4.8.3-4.el7.noarch @CentOS7 Dep-Install samba-common-libs-4.8.3-4.el7.x86_64 @CentOS7

1 0

Cephfs corruption(?) causing nfs-ganesha to "clients failing to respond to capability release" / "MDSs report slow requests"
by Marc Roos 26 Sep '19

26 Sep '19

Can it be that there is something wrong with the cephfs, that causes the nfs-ganesha to fail? I have the impression that when I do an rsync on only a specific nfs mount/cephfs tree. I am getting this error. All nfs mounts are inaccessible on that client. Other clients (with other mounts) are fine.

1 0

Re: Balancer active plan
by Massimo Sgaravatto 26 Sep '19

26 Sep '19

I think you should see this info in the mgr log (with debug mgr = 4/5) Cheers, Massimo On Thu, Sep 26, 2019 at 3:43 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > Suddenly I have objects misplaced, I assume this is because of the > balancer being active. But > > 1. how can I see the currently/last executed plan of the balancer? > 2. when it was activated? > > > 7 active+remapped+backfill_wait > 7 active+remapped+backfilling > > > > > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. > F1 Outsourcing Development Sp. z o.o. > Poland > > t: +48 (0)124466845 > f: +48 (0)124466843 > e: marc(a)f1-outsourcing.eu > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Balancer active plan
by Marc Roos 26 Sep '19

26 Sep '19

Suddenly I have objects misplaced, I assume this is because of the balancer being active. But 1. how can I see the currently/last executed plan of the balancer? 2. when it was activated? 7 active+remapped+backfill_wait 7 active+remapped+backfilling - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. F1 Outsourcing Development Sp. z o.o. Poland t: +48 (0)124466845 f: +48 (0)124466843 e: marc(a)f1-outsourcing.eu

1 0

CephFS deleted files' space not reclaimed
by Josh Haft 26 Sep '19

26 Sep '19

Hi, I've been migrating data from one EC pool to another EC pool: two directories are mounted with ceph.dir.layout.pool file attribute set appropriately, then rsync from old to new and finally, delete the old files. I'm using the kernel client to do this. While the removed files are no longer present on the filesystem, they still appear to be accounted for via "ceph df". When I tally up the sizes reported by "ls -lh" on all subdirectories under the root CephFS using a FUSE client mount (except for those on the new EC pool), it totals just under 2PiB. However, "ceph df" shows the original EC pool as 2.5PiB used. I've copied + deleted approximately 545TiB so far, so it seems like the unlinked files aren't being fully released/purged. I've only observed the num_strays counter from "ceph daemon mds.$name perf dump" for a few days now since I first suspected an issue, but I've never seen it drop below roughly 310k. From other ML postings I've gathered that stat has something to do with files pending deletion, but I'm not positive. So far all I've done is restart the mds and mon daemons, which hasn't helped. What are the next steps for troubleshooting? I can turn up mds debug logging, but am not sure what to look for. Thanks for your help! Josh

2 3

Luminous 12.2.12 "clients failing to respond to capability release"
by Marc Roos 26 Sep '19

26 Sep '19

I am getting this error I have in two sessions[0] num_caps high ( I assume the error is about num_caps ). I am using a default luminous and a default centos7 with default kernel 3.10. Do I really still need to change to a not stock kernel to resolve this? I read this in posts of 2016 and 2019, yet I also saw some post on tuning the mds. [0] ceph daemon mds.c session ls { "id": 3634861, "num_leases": 0, "num_caps": 2183, "state": "open", "request_load_avg": 181, "uptime": 2545057.775869, "replay_requests": 0, "completed_requests": 0, "reconnecting": false, { "id": 3644756, "num_leases": 0, "num_caps": 2080, "state": "open", "request_load_avg": 32, "uptime": 2545057.775869, "replay_requests": 0, "completed_requests": 0, "reconnecting": false,

1 4

Cephfs + docker
by Alex L 26 Sep '19

26 Sep '19

Hi, I am trying to figure out why my portainer and pi-hole in docker keeps getting broken databases. All other docker applications are working flawlessly but not these. I am running Ubuntu 18.04 + kernel ceph mount for the data directory. Have looked at how others do it, and they seem to all use the docker plugin for getting an RBD image but looking at those plugins many seem to not have been updated for years.. Thanks in advance, A

3 2

Nautilus dashboard: MDS performance graph doesn't refresh
by Eugen Block 26 Sep '19

26 Sep '19

Hi all, I regularly check the MDS performance graphs in the dashboard, especially the requests per second is interesting in my case. Since our upgrade to Nautilus the values in the activity column are still refreshed every 5 seconds (I believe), but the graphs are not refreshed since that upgrade anymore. I couldn't find anything in the tracker or the mailing list, can anyone comment on this? Thank you & best regards, Eugen

2 3

Re: Nautilus : ceph dashboard ssl not working
by Lenz Grimmer 26 Sep '19

26 Sep '19

Hi Miha, interesting observation, I don't think we've noticed this before. Would you mind submitting a bug report about this on our tracker, including these logs? https://tracker.ceph.com/projects/mgr/issues/new Thanks in advance! Lenz On 9/26/19 10:01 AM, Miha Verlic wrote: > On 24. 09. 19 14:53, Lenz Grimmer wrote: >> On 9/24/19 1:37 PM, Miha Verlic wrote: >> >>> I've got slightly different problem. After a few days of running fine, >>> dashboard stops working because it is apparently seeking for wrong >>> certificate file in /tmp. If I restart ceph-mgr it starts to work again. >> >> Does the restart trigger the creation of a similar-looking file in /tmp? >> I wonder if there's some kind of cron job that cleans up the /tmp >> directory every now and then... > > There is systemd-tmpfiles-clean.timer, but it ignores PrivateTmp folders. > > But as said, crt and key files do exist in tmp and according to > timestamps they were created when ceph-mgr daemon was started; it seems > to me like ceph-mgr simply starts looking for wrong filenames after a while. > > Going through ceph-mgr logs I'd say the problem is internal webserver is > restarted, but ceph-mgr is not notified about new crt/key files: > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Bus STOPPING > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 8443)) shut down > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Stopped thread '_TimeoutMonitor'. > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Bus STOPPED > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Bus STARTING > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Started monitor thread '_TimeoutMonitor'. > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: 2019-09-19 12:54:16.618 > 7f3a796e4700 -1 client.0 error registering admin socket command: (17) > File exists > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: 2019-09-19 12:54:16.618 > 7f3a796e4700 -1 client.0 error registering admin socket command: (17) > File exists > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: 2019-09-19 12:54:16.618 > 7f3a796e4700 -1 client.0 error registering admin socket command: (17) > File exists > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: 2019-09-19 12:54:16.618 > 7f3a796e4700 -1 client.0 error registering admin socket command: (17) > File exists > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: 2019-09-19 12:54:16.618 > 7f3a796e4700 -1 client.0 error registering admin socket command: (17) > File exists > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Serving on :::8443 > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Bus STARTED -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 247165 (AG Nürnberg)

1 0

how many monitor should to deploy in a 1000+ osd cluster
by 展荣臻（信泰） 26 Sep '19

26 Sep '19

hi all: I have a production cluster, and it had 24 hosts (528 osds,3mons) at a former. Now we want to add 36 hosts so the osd increase to 1320 . does the monitor need to increase?how many numbers of monitor node is recommended? Another question is which monitor does monclient commnuicate with? And how it decide? Any suggestions are welcome!

7 7

Re: download.ceph.com repository changes
by Alfredo Deza 25 Sep '19

25 Sep '19

Reviving this old thread. I still think this is something we should consider as users still experience problems: * Impossible to 'pin' to a version. User installs 14.2.0 and 4 months later they add other nodes but version moved to 14.2.2 * Impossible to use a version that is not what the latest is (e.g. if someone doesn't need the release from Monday, but wants the one from 6 months ago), similar to the above * When a release is underway, the repository breaks because syncing packages takes hours. The operation is not atomic. * It is not currently possible to "remove" a bad release, in the past, this means cutting a new release as soon as possible, which can take days The latest issue (my fault!) was to cut a release and get the packages out without communicating with the release manager, which caused users to note there is a new version *as soon as it was up* vs, a process that could've not touched the 'latest' url until the announcement goes out. If you have been affected by any of these issues (or others I didn't come up with), please let us know in this thread so that we can find some common ground and try to improve the process. Thanks! On Tue, Jul 24, 2018 at 10:38 AM Alfredo Deza <adeza(a)redhat.com> wrote: > > Hi all, > > After the 12.2.6 release went out, we've been thinking on better ways > to remove a version from our repositories to prevent users from > upgrading/installing a known bad release. > > The way our repos are structured today means every single version of > the release is included in the repository. That is, for Luminous, > every 12.x.x version of the binaries is in the same repo. This is true > for both RPM and DEB repositories. > > However, the DEB repos don't allow pinning to a given version because > our tooling (namely reprepro) doesn't construct the repositories in a > way that this is allowed. For RPM repos this is fine, and version > pinning works. > > To remove a bad version we have to proposals (and would like to hear > ideas on other possibilities), one that would involve symlinks and the > other one which purges the known bad version from our repos. > > *Symlinking* > When releasing we would have a "previous" and "latest" symlink that > would get updated as versions move forward. It would require > separation of versions at the URL level (all versions would no longer > be available in one repo). > > The URL structure would then look like: > > debian/luminous/12.2.3/ > debian/luminous/previous/ (points to 12.2.5) > debian/luminous/latest/ (points to 12.2.7) > > Caveats: the url structure would change from debian-luminous/ to > prevent breakage, and the versions would be split. For RPMs it would > mean a regression if someone is used to pinning, for example pinning > to 12.2.2 wouldn't be possible using the same url. > > Pros: Faster release times, less need to move packages around, and > easier to remove a bad version > > > *Single version removal* > Our tooling would need to go and remove the known bad version from the > repository, which would require to rebuild the repository again, so > that the metadata is updated with the difference in the binaries. > > Caveats: time intensive process, almost like cutting a new release > which takes about a day (and sometimes longer). Error prone since the > process wouldn't be the same (one off, just when a version needs to be > removed) > > Pros: all urls for download.ceph.com and its structure are kept the same.

9 11

Ceph NIC partitioning (NPAR)
by Adrien Georget 25 Sep '19

25 Sep '19

Hi, I need your advice about the following setup. Currently, we have a Ceph nautilus cluster used by Openstack Cinder with single NIC in 10Gbps on OSD hosts. We will upgrade the cluster by adding 7 new hosts dedicated to Nova/Glance and we would like to add a cluster network to isolate replication and recovery traffic. For now, it's not possible to add a second NIC and FC so we are thinking about enabling DELL NPAR [1] which allows splitting a single physical NIC in 2 logical NICs (1 for public network and 1 for Cluster network). We can set max and min bandwidth and implement dynamic bandwidth balancing for NPAR to get the appropriate bandwidth when Ceph need it (default alloc is 66% for cluster network and 34% for public network). Any experiences with this kind of configuration? Do you see any disadvantages doing this? And one question, if we put this in production, adding cluster network value in ceph.conf and restarting each OSD is enough for Ceph? Best, Adrien [1] https://www.dell.com/support/article/fr/fr/frbsdt1/how12596/how-npar-works?…

2 1

Announcing Ceph Buenos Aires 2019 on Oct 16th at Museo de Informatica
by Victoria Martinez de la Cruz 25 Sep '19

25 Sep '19

Hi all, I'm happy to announce that next Oct 16th we will have the Ceph Day Argentina in Buenos Aires. The event will be held in the Museo de Informatica de Argentina, so apart from hearing the latest features from core developers, real use cases from our users and usage experiences from customers and partners, you will be able to enjoy and contribute to this fantastic museum that holds a great collection of vintage hardware. If you are a local and/or you are in the area, we really hope you can join us! CFP is open in https://forms.zohopublic.com/thingee/form/CephDayArgentina2019/formperma/yf… Ceph Day Buenos Aires site is available in https://ceph.io/cephdays/ceph-day-argentina-2019/ Cheers, Victoria

1 0

Wrong %USED and MAX AVAIL stats for pool
by nalexandrov＠innologica.com 25 Sep '19

25 Sep '19

Hi everyone, We are running Nautilus 14.2.2 with 6 nodes and a total of 44 OSDs, all are 2TB spinning disks. # ceph osd count-metadata osd_objectstore "bluestore": 44 # ceph osd pool get one size size: 3 # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 80 TiB 33 TiB 47 TiB 47 TiB 58.26 TOTAL 80 TiB 33 TiB 47 TiB 47 TiB 58.26 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL one 2 15 TiB 4.06M 47 TiB 68.48 7.1 TiB bench 5 250 MiB 67 250 MiB 0 21 TiB Why pool's stats are showing incorrect values for %USED and MAX AVAIL? They should be much bigger. The first 24 OSDs was created on jewell release and the osd_objectstore was 'filestore'. While we were with mimic release, we added 20 more 'bluestore' OSDs. The first 24 was destroyed and recreated as 'bluestore'. After the upgrade from mimic release, all the OSD's was updated with ceph-bluestore-tool repair. The incorrect values appeared after the upgrade from 14.2.1 to 14.2.2. Any help will be appreciated :) BR, NAlexandrov

2 1

OSD rebalancing issue - should drives be distributed equally over all nodes
by Thomas 25 Sep '19

25 Sep '19

Hi, I'm facing several issues with my ceph cluster (2x MDS, 6x ODS). Here I would like to focus on the issue with pgs backfill_toofull. I assume this is related to the fact that the data distribution on my OSDs is not balanced. This is the current ceph status: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 MDSs report slow metadata IOs 78 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 2 pgs inactive, 2 pgs peering Degraded data redundancy: 304136/153251211 objects degraded (0.198%), 57 pgs degraded, 57 pgs undersized Degraded data redundancy (low space): 265 pgs backfill_toofull 3 pools have too many placement groups 74 slow requests are blocked > 32 sec 80 stuck requests are blocked > 4096 sec services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 98m) mgr: ld5505(active, since 3d), standbys: ld5506, ld5507 mds: pve_cephfs:1 {0=ld3976=up:active} 1 up:standby osd: 368 osds: 368 up, 367 in; 302 remapped pgs data: pools: 5 pools, 8868 pgs objects: 51.08M objects, 195 TiB usage: 590 TiB used, 563 TiB / 1.1 PiB avail pgs: 0.023% pgs not active 304136/153251211 objects degraded (0.198%) 1672190/153251211 objects misplaced (1.091%) 8564 active+clean 196 active+remapped+backfill_toofull 57 active+undersized+degraded+remapped+backfill_toofull 35 active+remapped+backfill_wait 12 active+remapped+backfill_wait+backfill_toofull 2 active+remapped+backfilling 2 peering io: recovery: 18 MiB/s, 4 objects/s Currently I'm using 6 OSD nodes. Node A 48x 1.6TB HDD Node B 48x 1.6TB HDD Node C 48x 1.6TB HDD Node D 48x 1.6TB HDD Node E 48x 7.2TB HDD Node F 48x 7.2TB HDD Question: Is it advisable to distribute the drives equally over all nodes? If yes, how should this be executed w/o ceph disruption? Regards Thomas

3 5

verify_upmap number of buckets 5 exceeds desired 4
by Eric Dold 25 Sep '19

25 Sep '19

Hello, I'm running ceph 14.2.3 on six hosts with each four osds. I did recently upgrade this from four hosts. The cluster is running fine. But i get this in my logs: Sep 11 11:02:41 ceph1 ceph-mon[1333]: 2019-09-11 11:02:41.953 7f26023a6700 -1 verify_upmap number of buckets 5 exceeds desired 4 Sep 11 11:02:41 ceph1 ceph-mon[1333]: 2019-09-11 11:02:41.953 7f26023a6700 -1 verify_upmap number of buckets 5 exceeds desired 4 Sep 11 11:02:41 ceph1 ceph-mon[1333]: 2019-09-11 11:02:41.953 7f26023a6700 -1 verify_upmap number of buckets 5 exceeds desired 4 It looks like the balancer is not doing any work. Here are some infos about the cluster: ceph1 ~ # ceph osd crush rule ls replicated_rule cephfs_ec ceph1 ~ # ceph osd crush rule dump replicated_rule { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ceph1 ~ # ceph osd crush rule dump cephfs_ec { "rule_id": 1, "rule_name": "cephfs_ec", "ruleset": 1, "type": 3, "min_size": 8, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "choose_indep", "num": 4, "type": "host" }, { "op": "choose_indep", "num": 2, "type": "osd" }, { "op": "emit" } ] } ceph1 ~ # ceph osd erasure-code-profile ls default isa_62 ceph1 ~ # ceph osd erasure-code-profile get default k=2 m=1 plugin=jerasure technique=reed_sol_van ceph1 ~ # ceph osd erasure-code-profile get isa_62 crush-device-class= crush-failure-domain=osd crush-root=default k=6 m=2 plugin=isa technique=reed_sol_van The idea with four hosts was that the ec profile should take two osds on each host for the eight buckets. Now with six hosts i guess two hosts will have tow buckets on two osds and four hosts will have each one bucket for a piece of data. Any idea how to resolve this? Regards Eric

1 2

configuration of Ceph-ISCSI gateway
by Gesiel Galvão Bernardes 24 Sep '19

24 Sep '19

Hi everyone, I'm configurating ISCSI gateway in Ceph Mimic (13.2.6) using ceph manual: https://docs.ceph.com/docs/mimic/rbd/iscsi-target-cli/ But i stopped in this problem: In manual says: "Set the client’s CHAP username to myiscsiusername and password to myiscsipassword: > /iscsi-target...at:rh7-client> auth chap=myiscsiusername/myiscsipassword" But I receive this response: /iscsi-target...at:rh7-client> auth chap=myiscsitest/myiscsitestpasswd Unexpected keyword parameter 'chap'. The options disponibles are: /iscsi-target...at:rh7-client> auth ? To set authentication, specify username=<user> password=<password> [mutual_username]=<user> [mutual_password]=<password> But if configure as asks: auth username=myiscsitest password=myiscsitestpasswd Failed to update the client's auth: Invalid password I tried with high password complexibility, but the problem persists. My questions: - How is the correct mode for configure authentication? - How contribute for update of documentation? A bug report has opened* for broken information of instalation of ceph-iscsi-gw, but was closed without update of documentation:https://github.com/ceph/ceph-ansible/issues/2707 Regards Gesiel Bernardeds

2 1

RGW orphaned shadow objects
by P. O. 24 Sep '19

24 Sep '19

Hi All, I have a question about "orphaned" objects in default.rgw.buckets.data pool. Few days ago i ran "radosgw-admin orphans find ..." [dc-1 root@mon-1 tmp]$ radosgw-admin orphans list-jobs [ "orphans-find-1" ] Today I checked the result. I listed orphaned objects by command: $# for i in `rados -p default.rgw.log ls |grep orphan.scan.orphans-find-1.rados`; do rados -p default.rgw.log listomapkeys $i; done > orphaned_objects.txt There are a lot: xxx.__shadow_.yyy objects. Is it possible to check if these __shadow_ are orphaned (can be removed) or belongs to any good object? How to check if a shadow object is still in use and what object it belongs to? Some of them are very old. For example: default.rgw.buckets.data/23a033d8-1146-2345-9f94-81383220c334.3130618.2__shadow_.-GOuvdWROljudUgTgq6u6wRR-lHoxU0_1 mtime 2017-08-16 04:01:49.000000, size 4194304 default.rgw.buckets.data/23a033d8-1146-2345-9f94-81383220c334.3130618.2__shadow_.-GOuvdWROljudUgTgq6u6wRR-lHoxU0_2 mtime 2017-08-16 04:01:49.000000, size 4194304 default.rgw.buckets.data/23a033d8-1146-2345-9f94-81383220c334.3130618.2__shadow_.-GOuvdWROljudUgTgq6u6wRR-lHoxU0_4 mtime 2017-08-16 04:01:49.000000, size 4194304 default.rgw.buckets.data/23a033d8-1146-2345-9f94-81383220c334.3130618.2__shadow_.-GOuvdWROljudUgTgq6u6wRR-lHoxU0_5 mtime 2017-08-16 04:01:49.000000, size 4194304 default.rgw.buckets.data/23a033d8-1146-2345-9f94-81383220c334.3130618.2__shadow_.-GOuvdWROljudUgTgq6u6wRR-lHoxU0_6 mtime 2017-08-16 04:01:49.000000, size 4194304 Best regards, PO

2 1

Health error: 1 MDSs report slow metadata IOs, 1 MDSs report slow requests
by Thomas 24 Sep '19

24 Sep '19

Hi, ceph health reports 1 MDSs report slow metadata IOs 1 MDSs report slow requests This is the complete output of ceph -s: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 MDSs report slow metadata IOs 1 MDSs report slow requests 72 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 33 pgs inactive, 32 pgs peering Degraded data redundancy: 123285/153918525 objects degraded (0.080%), 27 pgs degraded, 27 pgs undersized Degraded data redundancy (low space): 116 pgs backfill_toofull 3 pools have too many placement groups 54 slow requests are blocked > 32 sec 179 stuck requests are blocked > 4096 sec services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 21h) mgr: ld5507(active, since 21h), standbys: ld5506, ld5505 mds: pve_cephfs:1 {0=ld3955=up:active} 1 up:standby osd: 368 osds: 368 up, 368 in; 140 remapped pgs data: pools: 6 pools, 8872 pgs objects: 51.31M objects, 196 TiB usage: 591 TiB used, 561 TiB / 1.1 PiB avail pgs: 0.372% pgs not active 123285/153918525 objects degraded (0.080%) 621911/153918525 objects misplaced (0.404%) 8714 active+clean 90 active+remapped+backfill_toofull 26 active+undersized+degraded+remapped+backfill_toofull 16 peering 16 remapped+peering 7 active+remapped+backfill_wait 1 activating 1 active+recovery_wait+degraded 1 active+recovery_wait+undersized+remapped In the log I find these relevant entries: 2019-09-24 13:24:37.073695 mds.ld3955 [WRN] 2 slow requests, 0 included below; oldest blocked for > 18618.873983 secs 2019-09-24 13:24:42.073757 mds.ld3955 [WRN] 2 slow requests, 0 included below; oldest blocked for > 18623.874055 secs 2019-09-24 13:24:47.073852 mds.ld3955 [WRN] 2 slow requests, 0 included below; oldest blocked for > 18628.874149 secs 2019-09-24 13:24:52.073941 mds.ld3955 [WRN] 2 slow requests, 0 included below; oldest blocked for > 18633.874237 secs 2019-09-24 13:24:57.074073 mds.ld3955 [WRN] 2 slow requests, 0 included below; oldest blocked for > 18638.874354 secs 2019-09-24 13:25:02.074118 mds.ld3955 [WRN] 2 slow requests, 0 included below; oldest blocked for > 18643.874415 secs Cephfs is residing on a pool "hdd" with dedicated HDDs (4x 17 1.6TB). This pool is used for RBDs, too. Question: How can I identify the 2 slow requests? And how can I kill these requests? Regards Thomas

4 3

How to reduce or control memory usage during recovery?
by Amudhan P 24 Sep '19

24 Sep '19

Hi, I am using ceph mimic in a small test setup using the below configuration. OS: ubuntu 18.04 1 node running (mon,mds,mgr) + 4 core cpu and 4GB RAM and 1 Gb lan 3 nodes each having 2 osd's, disks are 2TB + 2 core cpu and 4G RAM and 1 Gb lan 1 node acting as cephfs client + 2 core cpu and 4G RAM and 1 Gb lan configured cephfs_metadata_pool (3 replica) and cephfs_data_pool erasure 2+1. When running a script doing multiple folders creation ceph started throwing error late IO due to high metadata workload. once after folder creation complete PG's degraded and I am waiting for PG to complete recovery but my OSD's starting to crash due to OOM and restarting after some time. Now my question is I can wait for recovery to complete but how do I stop OOM and OSD crash? basically want to know the way to control memory usage during recovery and make it stable. I have also set very low PG metadata_pool 8 and data_pool 16. I have already set "mon osd memory target to 1Gb" and I have set max-backfill from 1 to 8. Attached msg from "kern.log" from one of the node and snippet of error msg in this mail. ---------error msg snippet ---------- -bash: fork: Cannot allocate memory Sep 18 19:01:57 test-node1 kernel: [341246.765644] msgr-worker-0 invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Sep 18 19:02:00 test-node1 kernel: [341246.765645] msgr-worker-0 cpuset=/ mems_allowed=0 Sep 18 19:02:00 test-node1 kernel: [341246.765650] CPU: 1 PID: 1737 Comm: msgr-worker-0 Not tainted 4.15.0-45-generic #48-Ubuntu Sep 18 19:02:02 test-node1 kernel: [341246.765833] Out of memory: Kill process 1727 (ceph-osd) score 489 or sacrifice child Sep 18 19:02:03 test-node1 kernel: [341246.765919] Killed process 1727 (ceph-osd) total-vm:3483844kB, anon-rss:1992708kB, file-rss:0kB, shmem-rss:0kB Sep 18 19:02:03 test-node1 kernel: [341246.899395] oom_reaper: reaped process 1727 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Sep 18 22:09:57 test-node1 kernel: [352529.433155] perf: interrupt took too long (4965 > 4938), lowering kernel.perf_event_max_sample_rate to 40250 regards Amudhan

5 8

Re: Nautilus : ceph dashboard ssl not working
by Volker Theile 24 Sep '19

24 Sep '19

https://www.thegeekdiary.com/centos-rhel-67-why-the-files-in-tmp-directory-… Am 24.09.19 um 14:53 schrieb Lenz Grimmer: > On 9/24/19 1:37 PM, Miha Verlic wrote: > >> I've got slightly different problem. After a few days of running fine, >> dashboard stops working because it is apparently seeking for wrong >> certificate file in /tmp. If I restart ceph-mgr it starts to work again. > Does the restart trigger the creation of a similar-looking file in /tmp? > I wonder if there's some kind of cron job that cleans up the /tmp > directory every now and then... > > Lenz > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Volker Theile Software Engineer | Ceph | openATTIC SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany GF: Felix Imendörffer, HRB 247165 (AG München) Phone: +49 173 5876879 E-Mail: vtheile(a)suse.com

1 0

Configuration of Ceph-ISCSI gateway
by Gesiel Galvão Bernardes 24 Sep '19

24 Sep '19

Hi everyone, I'm configurating ISCSI gateway in Ceph Mimic (13.2.6) using ceph manual: https://docs.ceph.com/docs/mimic/rbd/iscsi-target-cli/ But i stopped in this problem: In manual says: "Set the client’s CHAP username to myiscsiusername and password to myiscsipassword: > /iscsi-target...at:rh7-client> auth chap=myiscsiusername/myiscsipassword" But I receive this response: /iscsi-target...at:rh7-client> auth chap=myiscsitest/myiscsitestpasswd Unexpected keyword parameter 'chap'. The options disponibles are: /iscsi-target...at:rh7-client> auth ? To set authentication, specify username=<user> password=<password> [mutual_username]=<user> [mutual_password]=<password> But if configure as asks: auth username=myiscsitest password=myiscsitestpasswd Failed to update the client's auth: Invalid password I tried with high password complexibility, but the problem persists. My questions: - How is the correct mode for configure authentication? - How contribute for update of documentation? A bug report has opened* for broken information of instalation of ceph-iscsi-gw, but was closed without update of documentation:https://github.com/ceph/ceph-ansible/issues/2707 Regards Gesiel Bernardeds

1 0

rados bench performance in nautilus
by 徐蕴 24 Sep '19

24 Sep '19

Hi ceph experts, I deployed Nautilus (v14.2.4) and Luminous (v12.2.11) on the same hardware, and made a rough performance comparison. The result seems Luminous is much better, which is unexpected. My setup: 3 servers, each has 3 HDD OSDs, 1 SSD as DB, two separated 1G network for cluster and public. Pool test has 32 pg and pop numbers, replicated size is 3. Using "rados -p bench 80 write” to measure write performance. The result: Luminous: Average IOPS 36 Nautilus: Average IOPS 28 Is the difference considered valid for Nautilus? Br, Xu Yun

3 6

Re: unsubscribe
by Wesley Peng 24 Sep '19

24 Sep '19

As the signature shows, please send an email to ceph-users-leave(a)ceph.io for unsubscribing. hou guanghua wrote: > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Creating a monmap with V1 & V2 using monmaptool
by Ricardo Dias 24 Sep '19

24 Sep '19

Hi Alberto, Did you tried with the "--addv" option? Here's an example: monmaptool --addv <mon.id> [v2:<ip_address>:<v2_port>,v1:<ip_address>:<v1_port>] Cheers, Ricardo Dias On 23/09/19 16:07, Corona, Alberto wrote: > Hi folks, > > While practicing some disaster recovery I noticed that it currently seems impossible to add both a v1 and v2 monitor to a monmap using monmaptool. Is there any way to create a monmap manually to include both protocol versions to a monmap? Currently on Ceph version 14.2.4 > > > - Alberto Corona > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >

1 0

unsubscribe
by hou guanghua 24 Sep '19

24 Sep '19

1 0

unsubscribe
by hou guanghua 24 Sep '19

24 Sep '19

1 0

cache tiering or bluestore partitions
by Shawn A Kwang 23 Sep '19

23 Sep '19

We are planning our ceph architecture and I have a question: How should NVMe drives be used when our spinning storage devices use bluestore: 1. block WAL and DB partitions (https://docs.ceph.com/docs/nautilus/rados/configuration/bluestore-config-re…) 2. Cache tier (https://docs.ceph.com/docs/nautilus/rados/operations/cache-tiering/) 3. Something else? Hardware- Each node has: 3x 8 TB HDD 1x 450 GB NVMe drive 192 GB RAM 2x Xeon CPUs (24 cores total) I plan to have three OSD daemons running on the node. There are 95 nodes total with the same hardware. Use Case: The plan is create cephfs and use this filesystem to store people's home directories and data. I anticipate more read operations than writes. Regarding cache tiering: The online documentation says cache tiering will often degrade performance. But when I read various threads on this ML there do seem to be people using cache tiering with success. I do see that it is heavily dependent upon one's use-case. In 2019 is there any updated recommendations as to whether to use cache tiering? If there is a third suggestion that people have I would be interested in hearing it. Thanks in advance. Sincerely, Shawn Kwang

2 3

Re: Seemingly unbounded osd_snap keys in monstore. Normal? Expected?
by Sage Weil 23 Sep '19

23 Sep '19

On Mon, 23 Sep 2019, Koebbe, Brian wrote: > Thanks Sage! > ceph osd dump: https://pastebin.com/raw/zLPz9DQg > > > ceph-monstore-tool /var/lib/ceph/mon/ceph-ufm03 dump-keys |grep osd_snap| cut -c-29 |uniq -c > 2 osd_snap / purged_snap_10_000 > 1 osd_snap / purged_snap_12_000 > 75 osd_snap / purged_snap_13_000 > 4 osd_snap / purged_snap_14_000 > 778106 osd_snap / purged_snap_15_000 > 861 osd_snap / purged_snap_1_0000 > 323 osd_snap / purged_snap_4_0000 > 88 osd_snap / purged_snap_5_0000 > 2 osd_snap / purged_snap_7_0000 > 2 osd_snap / purged_snap_8_0000 > 2 osd_snap / removed_epoch_10_0 > 1 osd_snap / removed_epoch_12_0 > 75 osd_snap / removed_epoch_13_0 > 4 osd_snap / removed_epoch_14_0 > 2316417 osd_snap / removed_epoch_15_0 > 970 osd_snap / removed_epoch_1_00 > 324 osd_snap / removed_epoch_4_00 > 89 osd_snap / removed_epoch_5_00 > 2 osd_snap / removed_epoch_7_00 > 2 osd_snap / removed_epoch_8_00 > 2 osd_snap / removed_snap_10_00 > 1 osd_snap / removed_snap_12_00 > 75 osd_snap / removed_snap_13_00 > 4 osd_snap / removed_snap_14_00 > 2720849 osd_snap / removed_snap_15_00 > 1161 osd_snap / removed_snap_1_000 > 379 osd_snap / removed_snap_4_000 > 89 osd_snap / removed_snap_5_000 > 2 osd_snap / removed_snap_7_000 > 2 osd_snap / removed_snap_8_000 Thanks! I opened a ticket at https://tracker.ceph.com/issues/42012. Can you do the aboe dump with the 'grep osd_snap' only and attach that to the bug? This code has reworekd/improved in master, but I'm not sure the behavior of keeping a full record of past span deletions was changed, so we may need to make further improvements for octopus. Thanks! sage > ________________________________ > From: Sage Weil <sage(a)newdream.net> > Sent: Monday, September 23, 2019 9:41 AM > To: Koebbe, Brian <koebbe(a)wustl.edu> > Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io>; dev(a)ceph.io <dev(a)ceph.io> > Subject: Re: [ceph-users] Seemingly unbounded osd_snap keys in monstore. Normal? Expected? > > Hi, > > On Mon, 23 Sep 2019, Koebbe, Brian wrote: > > Our cluster has a little over 100 RBDs. Each RBD is snapshotted with a typical "frequently", hourly, daily, monthly type of schedule. > > A while back a 4th monitor was temporarily added to the cluster that took hours to synchronize with the other 3. > > While trying to figure out why that addition took so long, we discovered that our monitors have what seems like a really large number of osd_snap keys: > > > > ceph-monstore-tool /var/lib/ceph/mon/xxxxxx dump-keys |awk '{print $1}'|uniq -c > > 153 auth > > 2 config > > 10 health > > 1441 logm > > 3 mdsmap > > 313 mgr > > 1 mgr_command_descs > > 3 mgr_metadata > > 163 mgrstat > > 1 mkfs > > 323 mon_config_key > > 1 mon_sync > > 6 monitor > > 1 monitor_store > > 32 monmap > > 120 osd_metadata > > 1 osd_pg_creating > > 5818618 osd_snap > > 41338 osdmap > > 754 paxos > > > > A few questions: > > > > Could this be the cause of the slow addition/synchronization? > > Probably! > > > Is what looks like an unbounded number of osd_snaps expected? > > Maybe. Can you send me the output of 'ceph osd dump'? Also, if you don't > mind doing the dump above and grepping out just the osd_snap keys, so I > can see what they look like and if they match the osd map contents? > > Thanks! > sage > > > > If trimming/compacting them would help, how would one do that? > > > > Thanks, > > Brian > > > > ________________________________ > > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. >

2 1

Re: Seemingly unbounded osd_snap keys in monstore. Normal? Expected?
by Sage Weil 23 Sep '19

23 Sep '19

Hi, On Mon, 23 Sep 2019, Koebbe, Brian wrote: > Our cluster has a little over 100 RBDs. Each RBD is snapshotted with a typical "frequently", hourly, daily, monthly type of schedule. > A while back a 4th monitor was temporarily added to the cluster that took hours to synchronize with the other 3. > While trying to figure out why that addition took so long, we discovered that our monitors have what seems like a really large number of osd_snap keys: > > ceph-monstore-tool /var/lib/ceph/mon/xxxxxx dump-keys |awk '{print $1}'|uniq -c > 153 auth > 2 config > 10 health > 1441 logm > 3 mdsmap > 313 mgr > 1 mgr_command_descs > 3 mgr_metadata > 163 mgrstat > 1 mkfs > 323 mon_config_key > 1 mon_sync > 6 monitor > 1 monitor_store > 32 monmap > 120 osd_metadata > 1 osd_pg_creating > 5818618 osd_snap > 41338 osdmap > 754 paxos > > A few questions: > > Could this be the cause of the slow addition/synchronization? Probably! > Is what looks like an unbounded number of osd_snaps expected? Maybe. Can you send me the output of 'ceph osd dump'? Also, if you don't mind doing the dump above and grepping out just the osd_snap keys, so I can see what they look like and if they match the osd map contents? Thanks! sage > If trimming/compacting them would help, how would one do that? > > Thanks, > Brian > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. >

2 1

Activate Cache Tier on Running Pools
by Eikermann, Robert 23 Sep '19

23 Sep '19

Hi, I'm using Ceph in combination with Openstack. For the "VMs" Pool I'd like to enable writeback caching tier, like described here: https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ . Should it be possible to do that on a running pool? I tried to do so and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly filesystems. No errors were shown in ceph (but also no traffic arrived after enabling the cache tier). Removing the cache tier , rebooting the VMs and doing a filesystemcheck repaired everything. Best Robert

6 9

Seemingly unbounded osd_snap keys in monstore. Normal? Expected?
by Koebbe, Brian 23 Sep '19

23 Sep '19

Our cluster has a little over 100 RBDs. Each RBD is snapshotted with a typical "frequently", hourly, daily, monthly type of schedule. A while back a 4th monitor was temporarily added to the cluster that took hours to synchronize with the other 3. While trying to figure out why that addition took so long, we discovered that our monitors have what seems like a really large number of osd_snap keys: ceph-monstore-tool /var/lib/ceph/mon/xxxxxx dump-keys |awk '{print $1}'|uniq -c 153 auth 2 config 10 health 1441 logm 3 mdsmap 313 mgr 1 mgr_command_descs 3 mgr_metadata 163 mgrstat 1 mkfs 323 mon_config_key 1 mon_sync 6 monitor 1 monitor_store 32 monmap 120 osd_metadata 1 osd_pg_creating 5818618 osd_snap 41338 osdmap 754 paxos A few questions: Could this be the cause of the slow addition/synchronization? Is what looks like an unbounded number of osd_snaps expected? If trimming/compacting them would help, how would one do that? Thanks, Brian ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

1 0

Errors handle_connect_reply_2 connect got BADAUTHORIZER
by Thomas 23 Sep '19

23 Sep '19

Hi, this is another issue that I'm facing with my current Ceph cluster (2x MDS, 6x ODS). When I try to delete a RBD image the CLI is flooded with error messages: root@ld3955:# rbd rm hdd/vm-304-disk-0 Removing image: 3% complete...2019-09-23 14:25:01.954 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb13403f1d0 0x7fb1340416e0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.954 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144076720 0x7fb144078c00 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.958 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb14407ca70 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.958 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144076720 0x7fb144078ba0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 6% complete...2019-09-23 14:25:01.958 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079460 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 7% complete...2019-09-23 14:25:01.958 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144076720 0x7fb144078ba0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.958 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079460 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.958 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144076720 0x7fb144078ba0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 8% complete...2019-09-23 14:25:01.958 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079460 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 9% complete...2019-09-23 14:25:01.962 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079dc0 0x7fb144078730 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 10% complete...2019-09-23 14:25:01.962 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079440 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.962 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079da0 0x7fb144078730 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 12% complete...2019-09-23 14:25:01.962 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079440 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.962 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079da0 0x7fb144078730 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 13% complete...2019-09-23 14:25:01.962 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079440 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 14% complete...2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c80 0x7fb144078730 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 15% complete...2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c60 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 16% complete...2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 17% complete...2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c60 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c60 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 19% complete...2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c60 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 20% complete...2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c60 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 21% complete...2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 22% complete...2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c60 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 24% complete...2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c40 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.970 7fb15bfcb700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb1340ba610 0x7fb1340bca70 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 25% complete...2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079350 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407ea30 0x7fb144081410 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15bfcb700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb144079c40 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079350 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 26% complete...2019-09-23 14:25:01.974 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb14407ea30 0x7fb144080fc0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c40 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407ea30 0x7fb144082c20 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb14407a490 0x7fb144079350 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079310 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb144079bf0 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 27% complete...2019-09-23 14:25:01.974 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb14407a490 0x7fb144079310 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407ea30 0x7fb144082c20 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 28% complete...2019-09-23 14:25:01.974 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb144079bb0 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079310 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb14407ea30 0x7fb144082c20 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 29% complete...2019-09-23 14:25:01.978 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079bb0 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER How can I fix this error? THX

1 0

vfs_ceph and permissions
by ceph-users＠dxps31.33mail.com 23 Sep '19

23 Sep '19

Hi ceph-users, I'm having problems with ACLs and vfs_ceph. I'm not sure this is the right list for those questions, please direct me elsewhere or just tell me to bugger off if I'm in the wrong place. I am running a Samba 4.7 domain controller connecting to a nautilus ceph cluster. I am using proxmox for the ceph cluster and Zentyal for managing the samba domain. I am not able to set the permissions for the share within Computer Management My smb.conf share settings block is: [data2] browseable = yes force create mode = 0660 force directory mode = 0660 valid users = @"Domain Users", @"Domain Admins", @"Domain Admins" read list = write list = @"Domain Users", @"Domain Admins" admin users = @"Domain Admins" vfs objects = acl_xattr full_audit full_audit:failure = connect opendir disconnect unlink mkdir rmdir open rename map acl inherit = yes store dos attributes = yes vfs objects = ceph path = /new/ kernel share modes = no ceph:config_file = /etc/ceph/ceph.conf ceph:user_id = admin (I know - I will shange the user from admin as soon as I have a working setup!) The settings are similar share (set up by the Zentyal UI) that allows me to set permissions correctly: [test] path = /home/samba/shares/test browseable = yes force create mode = 0660 force directory mode = 0660 valid users = @"Domain Admins", @"Domain Admins" read list = write list = @"Domain Admins" admin users = @"Domain Admins" vfs objects = acl_xattr full_audit recycle full_audit:failure = connect opendir disconnect unlink mkdir rmdir open rename When trying to, for instance, add full control permissions for a user in Computer Management I get "Failed to enumerate objects in the container. Access is denied." / "Unable to change permission changes [..] Access denied.". I am logged in as administrator when trying to change the permissions and this is (an extract) of the error in the samba.log [2019/09/07 15:57:14.656361, 2] ../source3/smbd/posix_acls.c:3008(set_canon_ace_list) set_canon_ace_list: sys_acl_set_file type file failed for file . (Operation not supported). [2019/09/07 15:57:14.656435, 3] ../source3/smbd/posix_acls.c:3093(convert_canon_ace_to_posix_perms) convert_canon_ace_to_posix_perms: Too many ACE entries for file . to convert to posix perms. [2019/09/07 15:57:14.656472, 3] ../source3/smbd/posix_acls.c:3986(set_nt_acl) set_nt_acl: failed to convert file acl to posix permissions for file .. [2019/09/07 15:57:14.656511, 3] ../source3/smbd/smb2_server.c:3139(smbd_smb2_request_error_ex) smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] || at ../source3/smbd/smb2_setinfo.c:132 [2019/09/07 15:57:18.042643, 3] ../source3/smbd/nttrans.c:2036(smbd_do_query_security_desc) smbd_do_query_security_desc: sd_size = 120. [2019/09/07 15:57:18.043643, 3] ../source3/smbd/nttrans.c:2036(smbd_do_query_security_desc) smbd_do_query_security_desc: sd_size = 76. [2019/09/07 15:57:18.045086, 2] ../source3/smbd/posix_acls.c:3008(set_canon_ace_list) set_canon_ace_list: sys_acl_set_file type file failed for file . (Operation not supported). [2019/09/07 15:57:18.045171, 3] ../source3/smbd/posix_acls.c:3093(convert_canon_ace_to_posix_perms) convert_canon_ace_to_posix_perms: Too many ACE entries for file . to convert to posix perms. [2019/09/07 15:57:18.045208, 3] ../source3/smbd/posix_acls.c:3986(set_nt_acl) set_nt_acl: failed to convert file acl to posix permissions for file .. [2019/09/07 15:57:18.045248, 3] ../source3/smbd/smb2_server.c:3139(smbd_smb2_request_error_ex) smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] || at ../source3/smbd/smb2_setinfo.c:132 [2019/09/07 15:57:19.016012, 3] ../source3/winbindd/winbindd_misc.c:395(winbindd_interface_version) [ 6733]: request interface version (version = 29) [2019/09/07 15:57:19.016349, 3] ../source3/winbindd/winbindd_misc.c:428(winbindd_priv_pipe_dir) Any pointers would be great!

3 5

V/v [Ceph] problem with delete object in large bucket
by tuan dung 23 Sep '19

23 Sep '19

Hi Ceph team, Can you explain for me how ceph deleting object work? I have a bucket with above 100M object (file size ~ 50KB). When I delete object for free space, speed of deleting object very slow (about ~ 30-33 objects /s). I wan to tuning performance of cluster but i do not clearly know how ceph do deleting objects? Thank you very much ----------------------------- Br, Dương Tuấn Dũng

2 2

Cannot start virtual machines KVM / LXC
by Thomas 23 Sep '19

23 Sep '19

Hi, here I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS). The issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds. All these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd This pool hdd is relatively small compared to the largest pool: hdb_backup root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B hdb_backup 589 TiB 51262212 0 153786636 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 B 0 B hdd 3.2 TiB 281884 6568 845652 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 B 0 B pve_cephfs_data 955 GiB 91832 0 275496 0 0 3038 2103 1021 MiB 102170 318 GiB 0 B 0 B pve_cephfs_metadata 486 MiB 62 0 186 0 0 7 860 1.4 GiB 12393 166 MiB 0 B 0 B total_objects 51635990 total_used 597 TiB total_avail 522 TiB total_space 1.1 PiB This is the current health status of the ceph cluster: cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 87 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 54 pgs inactive, 47 pgs peering, 1 pg stale Degraded data redundancy: 129598/154907946 objects degraded (0.084%), 33 pgs degraded, 33 pgs undersized Degraded data redundancy (low space): 322 pgs backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 1 pools have too many placement groups 21 slow requests are blocked > 32 sec services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby osd: 360 osds: 356 up, 356 in; 382 remapped pgs data: pools: 5 pools, 8868 pgs objects: 51.64M objects, 197 TiB usage: 597 TiB used, 522 TiB / 1.1 PiB avail pgs: 0.056% pgs unknown 0.553% pgs not active 129598/154907946 objects degraded (0.084%) 2211119/154907946 objects misplaced (1.427%) 8458 active+clean 298 active+remapped+backfill_toofull 29 remapped+peering 24 active+undersized+degraded+remapped+backfill_toofull 22 active+remapped+backfill_wait 17 peering 5 unknown 5 active+recovery_wait+undersized+degraded+remapped 3 active+undersized+degraded+remapped+backfill_wait 2 activating+remapped 1 active+clean+remapped 1 stale+peering 1 active+remapped+backfilling 1 active+recovering+undersized+remapped 1 active+recovery_wait+degraded io: client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr I believe the cluster is busy with rebalancing pool hdb_backup. I set the balance mode upmap recently after the 589TB data was written. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } In order to resolve the issue with pool hdd I started some investigation. First step was to install drivers for the NIC provided Mellanox. Then I configured some kernel parameters recommended <https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox. However this didn't fix the issue. In my opinion I must get rid of all "slow requests are blocked". When I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd. This means none of the disks belonging to pool hdb_backup is showing a comparable behaviour. Then I checked the running processes on the different OSD nodes; I use tool "glances" here. Here I can see single processes that are running for hours and consuming much CPU, e.g. 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S 41M 2K /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S 14M 3K /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph --setgroup ceph Similar processes are running on 4 OSD nodes. All processes have in common that the relevant OSD belongs to pool hdd. Furthermore glances gives me this alert: CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, ceph-osd, ceph-osd What can / should I do now? Kill the long running processes? Stop the relevant OSDs? Please advise? THX Thomas

4 9

Ceph FS not releasing space after file deletion
by Gustavo Tonini 23 Sep '19

23 Sep '19

Dear ceph users, we are using CEPH since Jewel and now (with Mimic) are facing a problem that is impeding us to use it appropriately. After deleting some big backup files, we have noticed that their space has not returned to the pool's free space. Considering the mentioned pool, du shows that we have a total of 247TB allocated in files and "ceph df" says we have 293TB of used space. How can we debug this problem and force ceph to release unallocated space? Gustavo.

5 9

Binding to cluster-addr
by 徐蕴 22 Sep '19

22 Sep '19

Hello Ceph Users, I deployed a ceph cluster (v14.2.1, using docker), the cluster status seems OK but the write performance tested by rados bench seems bad. When I check the network connection using netstat -nat, I found that there is no connection established using the cluster network interface. The OSD daemon entrypoint is something like this: /usr/bin/ceph-osd -f --public-addr 111.111.121.2 --cluster-addr 111.111.131.2 -i 1 But there is no connection established using the IP 111.111.131.2. Is it a problematic behavior? If yes, would you please give me some hints to debug this problem? Thank you! Xu Yun

1 1

Identify rbd snapshot
by Marc Roos 21 Sep '19

21 Sep '19

I have this error. I have found the rbd image with the block_name_prefix:1f114174b0dc51, how can identify what snapshot this is? (Is it a snapshot?) 2019-08-29 16:16:49.255183 7f9b3f061700 -1 log_channel(cluster) log [ERR] : deep-scrub 17.36 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head : expected clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 missing 2019-08-29 16:24:54.806912 7f9b3f061700 -1 log_channel(cluster) log [ERR] : 17.36 deep-scrub 1 errors

2 1

RGW backup to tape
by Robert LeBlanc 20 Sep '19

20 Sep '19

The question was posed, "What if we want to backup our RGW data to tape?" Anyone doing this? Any suggestions? We could probably just catch any PUT requests and queue them to be written to tape. Our dataset is so large, that traditional backup solutions don't seem feasible (GFS), so probably a single copy (or two copies on different tapes at the same time) when the object is created. Bonus points for being near-line. Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

3 3

HEALTH_WARN due to large omap object wont clear even after trim
by shubjero 20 Sep '19

20 Sep '19

Hey all, Yesterday our cluster went in to HEALTH_WARN due to 1 large omap object in the .usage pool (I've posted about this in the past). Last time we resolved the issue by trimming the usage log below the alert threshold but this time it seems like the alert wont clear even after trimming and (this time) disabling the usage log entirely. ceph health detail HEALTH_WARN 1 large omap objects LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool '.usage' Search the cluster log for 'Large omap object found' for more details. I've bounced ceph-mon, ceph-mgr, radosgw and even issued osd scrub on the two osd's that hold pg's for the .usage pool but the alert wont clear. It's been over 24 hours since I trimmed the usage log. Any suggestions? Jared Baker Cloud Architect, OICR

3 5

How to set timeout on Rados gateway request
by Hanyu Liu 20 Sep '19

20 Sep '19

Hi, We are looking for a way to set timeout on requests to rados gateway. If a request takes too long time, just kill it. 1. Is there a command that can set the timeout? 2. This parameter looks interesting. Can I know what the "open threads" means? rgw op thread timeout Description: The timeout in seconds for open threads. Type: Integer Default: 600 *(from https://docs.ceph.com/docs/nautilus/radosgw/config-ref/ <https://docs.ceph.com/docs/nautilus/radosgw/config-ref/>)* Thanks, Hanyu

2 1

handle_connect_reply_2 connect got BADAUTHORIZER when running ceph pg <id> query
by Thomas 20 Sep '19

20 Sep '19

Hi, ceph health status reports unknown objects. All objects reside on same osd.9 When I execute ceph pg <id> query I get this (endless) output: 2019-09-20 14:47:35.922 7f937144f700 0 --1- 10.97.206.91:0/2060489821 >> v1:10.97.206.93:7054/15812 conn(0x7f935407c120 0x7f935407b120 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER ^CTraceback (most recent call last): File "/usr/bin/ceph", line 1263, in <module> retval = main() File "/usr/bin/ceph", line 1179, in main prefix='get_command_descriptions') File "/usr/lib/python2.7/dist-packages/ceph_argparse.py", line 1459, in json_command inbuf, timeout, verbose) File "/usr/lib/python2.7/dist-packages/ceph_argparse.py", line 1329, in send_command_retry return send_command(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/ceph_argparse.py", line 1381, in send_command cluster.pg_command, pgid, cmd, inbuf, timeout=timeout) File "/usr/lib/python2.7/dist-packages/ceph_argparse.py", line 1311, in run_in_thread t.join(timeout=timeout) File "/usr/lib/python2.7/threading.py", line 951, in join self.__block.wait(delay) File "/usr/lib/python2.7/threading.py", line 359, in wait _sleep(delay) KeyboardInterrupt 2019-09-20 14:47:35.950 7f937144f700 0 --1- 10.97.206.91:0/2060489821 >> v1:10.97.206.93:7054/15812 conn(0x7f935407f4b0 0x7f935407b920 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-20 14:47:35.950 7f937144f700 0 --1- 10.97.206.91:0/2060489821 >> v1:10.97.206.93:7054/15812 conn(0x7f935407c120 0x7f935407b120 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-20 14:47:35.950 7f937144f700 0 --1- 10.97.206.91:0/2060489821 >> v1:10.97.206.93:7054/15812 conn(0x7f935407f4b0 0x7f935407b920 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER How can I fix this issue with pg / osd.9? THX

1 0

Re: ceph fs crashes on simple fio test
by Robert LeBlanc 20 Sep '19

20 Sep '19

On Tue, Sep 10, 2019 at 1:11 PM Frank Schilder <frans(a)dtu.dk> wrote: > Hi Robert, > > I have meta data on SSD (3xrep) and data on 8+2 EC on spinning disks, so > the speed difference is orders of magnitudes. Our usage is quite meta data > heavy, so this suits us well. In particular since EC pools are high > throughput with large IO sizes. > > As long as one uses fio with direct=1 (probably also if using sync=1 > and/or fsync=1), everything is fine and behaves as you describe. IOPs > fluctuate but adjust to media speed. No problems at all. > > As mentioned in my last update (I cut it out below), the destructive fio > command runs with direct=0 and neither sync=1 nor fsync=1. This test just > writes as fast as it can (to buffers) without waiting for acks. I would > expect that a ceph client would translate that to synced or direct IO, > which would be fine. > > But it doesn't. Instead, it pushes the IO also as fast as possible to the > cluster. I have seen 40kops write on the EC pool (on 100+ HDDs) that can > handle maybe 1kops write in total. The queues were constantly increasing at > an incredible rate (several hundred ops per second). I hope with the change > of cut_off=high that heartbeats will not get lost any more, but this will > still destabilize our ceph cluster quite dramatically. > Changing the cut_off to high will not allow heartbeats to not get lost (heartbeats have a priority far above the high mark). What cut_off = high does is put replication ops into the main queue instead of the strict priority queue. That way an OSD doesn't get DDOSed from it's peers and is never able to service it's own clients. When I did my fio testing, was on FireFly/Hammer and on RBD, so I can't talk specifically to newer versions and CephFS. We haven't had time to set up our test cluster, so I can't run benches at the moment. > My problem is not so much that such an IO pattern could occur in > reasonable software, but > - that someone might try just for fun, and that > - the number of 500+ clients might occasionally produce such a workload by > aggregation. > > I find it somewhat alarming that a storage system that promises data > integrity and reliability can be taken down with a publicly available > benchmark tool in a matter of a few dozen seconds by ordinary users. > Potentially with damaging effects. I guess something similar could be > achieved with a modified rogue client. > > I would expect that a storage cluster should have basic self-defence > mechanisms that prevent this kind of overload or DOS attack by throttling > clients with crazy IO requests. Are there any settings that can be enabled > to prevent this from happening? > ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 1

Re: ceph mdss keep on crashing after update to 14.2.3
by Sasha Litvak 20 Sep '19

20 Sep '19

Any chance for a fix soon? In 14.2.5 ? On Thu, Sep 19, 2019 at 8:44 PM Yan, Zheng <ukernel(a)gmail.com> wrote: > On Thu, Sep 19, 2019 at 11:37 PM Dan van der Ster <dan(a)vanderster.com> > wrote: > > > > You were running v14.2.2 before? > > > > It seems that that ceph_assert you're hitting was indeed added > > between v14.2.2. and v14.2.3 in this commit > > > https://github.com/ceph/ceph/commit/12f8b813b0118b13e0cdac15b19ba8a7e127730b > > > > There's a comment in the tracker for that commit which says the > > original fix was incomplete > > (https://tracker.ceph.com/issues/39987#note-5) > > > > So perhaps nautilus needs > > > https://github.com/ceph/ceph/pull/28459/commits/0a1e92abf1cfc8bddf526cbf5bc… > > ?? > > > > You are right. Sorry for the bug. For now, please got back to 14.2.2 > (just mds) or complie ceph-mds from source > > Yan, Zheng > > > Did you already try going back to v14.2.2 (on the MDS's only) ?? > > > > -- dan > > > > On Thu, Sep 19, 2019 at 4:59 PM Kenneth Waegeman > > <kenneth.waegeman(a)ugent.be> wrote: > > > > > > Hi all, > > > > > > I updated our ceph cluster to 14.2.3 yesterday, and today the mds are > crashing one after another. I'm using two active mds. > > > > > > I've made a tracker ticket, but I was wondering if someone else also > has seen this issue yet? > > > > > > -27> 2019-09-19 15:42:00.196 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865333:8887 lookup > #0x100166004d4/WindowsPhone-MSVC-CXX.cmake 2019-09-19 15:42:00.203132 > caller_uid=0, caller_gid=0{0,}) v4 > > > -26> 2019-09-19 15:42:00.196 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865372:5815 lookup > #0x20005a6eb3a/selectable.cpython-37.pyc 2019-09-19 15:42:00.204970 > caller_uid=0, caller_gid=0{0,}) v4 > > > -25> 2019-09-19 15:42:00.196 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865333:8888 lookup > #0x100166004d4/WindowsPhone.cmake 2019-09-19 15:42:00.206381 caller_uid=0, > caller_gid=0{0,}) v4 > > > -24> 2019-09-19 15:42:00.206 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865333:8889 lookup > #0x100166004d4/WindowsStore-MSVC-C.cmake 2019-09-19 15:42:00.209703 > caller_uid=0, caller_gid=0{0,}) v4 > > > -23> 2019-09-19 15:42:00.206 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865333:8890 lookup > #0x100166004d4/WindowsStore-MSVC-CXX.cmake 2019-09-19 15:42:00.213200 > caller_uid=0, caller_gid=0{0,}) v4 > > > -22> 2019-09-19 15:42:00.216 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865333:8891 lookup > #0x100166004d4/WindowsStore.cmake 2019-09-19 15:42:00.216577 caller_uid=0, > caller_gid=0{0,}) v4 > > > -21> 2019-09-19 15:42:00.216 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865333:8892 lookup > #0x100166004d4/Xenix.cmake 2019-09-19 15:42:00.220230 caller_uid=0, > caller_gid=0{0,}) v4 > > > -20> 2019-09-19 15:42:00.216 7f0369aeb700 2 mds.1.cache Memory > usage: total 4603496, rss 4167920, heap 323836, baseline 323836, 501 / > 1162471 inodes have caps, 506 caps, 0.00043528 caps per inode > > > -19> 2019-09-19 15:42:00.216 7f03652e2700 5 mds.1.log > _submit_thread 30520209420029~9062 : EUpdate scatter_writebehind [metablob > 0x1000bd8ac7b, 2 dirs] > > > -18> 2019-09-19 15:42:00.216 7f03652e2700 5 mds.1.log > _submit_thread 30520209429111~10579 : EUpdate scatter_writebehind [metablob > 0x1000bf26309, 9 dirs] > > > -17> 2019-09-19 15:42:00.216 7f03652e2700 5 mds.1.log > _submit_thread 30520209439710~2305 : EUpdate scatter_writebehind [metablob > 0x1000bf2745b.001*, 2 dirs] > > > -16> 2019-09-19 15:42:00.216 7f03652e2700 5 mds.1.log > _submit_thread 30520209442035~1845 : EUpdate scatter_writebehind [metablob > 0x1000c233753, 2 dirs] > > > -15> 2019-09-19 15:42:00.216 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865333:8893 lookup > #0x100166004d4/eCos.cmake 2019-09-19 15:42:00.223360 caller_uid=0, > caller_gid=0{0,}) v4 > > > -14> 2019-09-19 15:42:00.216 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865319:2381 lookup > #0x1001172f39d/microsoft-cp1251 2019-09-19 15:42:00.224940 caller_uid=0, > caller_gid=0{0,}) v4 > > > -13> 2019-09-19 15:42:00.226 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865333:8894 lookup > #0x100166004d4/gas.cmake 2019-09-19 15:42:00.226624 caller_uid=0, > caller_gid=0{0,}) v4 > > > -12> 2019-09-19 15:42:00.226 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865319:2382 readdir > #0x1001172f3d7 2019-09-19 15:42:00.228673 caller_uid=0, caller_gid=0{0,}) v4 > > > -11> 2019-09-19 15:42:00.226 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865333:8895 lookup > #0x100166004d4/kFreeBSD.cmake 2019-09-19 15:42:00.229668 caller_uid=0, > caller_gid=0{0,}) v4 > > > -10> 2019-09-19 15:42:00.226 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865333:8896 lookup > #0x100166004d4/syllable.cmake 2019-09-19 15:42:00.232746 caller_uid=0, > caller_gid=0{0,}) v4 > > > -9> 2019-09-19 15:42:00.236 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865333:8897 readdir > #0x10016601379 2019-09-19 15:42:00.240672 caller_uid=0, caller_gid=0{0,}) v4 > > > -8> 2019-09-19 15:42:00.236 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865356:3604574 readdir > #0x2000090d630 2019-09-19 15:42:00.241832 caller_uid=0, caller_gid=0{0,}) v4 > > > -7> 2019-09-19 15:42:00.266 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865356:3604575 readdir > #0x2000090d631 2019-09-19 15:42:00.272158 caller_uid=0, caller_gid=0{0,}) v4 > > > -6> 2019-09-19 15:42:00.326 7f03652e2700 5 mds.1.log > _submit_thread 30520209443900~3089 : EUpdate scatter_writebehind [metablob > 0x20005af5c63, 3 dirs] > > > -5> 2019-09-19 15:42:00.326 7f03652e2700 5 mds.1.log > _submit_thread 30520209447009~10579 : EUpdate scatter_writebehind [metablob > 0x1000bf26309, 9 dirs] > > > -4> 2019-09-19 15:42:00.326 7f03652e2700 5 mds.1.log > _submit_thread 30520209457608~2305 : EUpdate scatter_writebehind [metablob > 0x1000bf2745b.001*, 2 dirs] > > > -3> 2019-09-19 15:42:00.326 7f03652e2700 5 mds.1.log > _submit_thread 30520209459933~1030 : EUpdate check_inode_max_size [metablob > 0x20005af5c74, 1 dirs] > > > -2> 2019-09-19 15:42:00.326 7f036c2f0700 4 mds.1.server > handle_client_request client_request(client.37865372:5816 setattr size=1138 > #0x20005a6eb67 2019-09-19 15:42:00.333015 caller_uid=0, caller_gid=0{0,}) v4 > > > -1> 2019-09-19 15:42:00.336 7f036c2f0700 -1 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.3/rpm/el7/BUILD/ceph-14.2.3/src/mds/MDCache.cc: > In function 'CInode* MDCache::cow_inode(CInode*, snapid_t)' thread > 7f036c2f0700 time 2019-09-19 15:42:00.333567 > > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.3/rpm/el7/BUILD/ceph-14.2.3/src/mds/MDCache.cc: > 1498: FAILED ceph_assert(!lock->get_num_wrlocks()) > > > > > > ceph version 14.2.3 (0f776cf838a1ae3130b2b73dc26be9c95c6ccc39) > nautilus (stable) > > > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x14a) [0x7f0375773ac2] > > > 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char > const*, char const*, ...)+0) [0x7f0375773c90] > > > 3: (()+0x1ee48d) [0x55a7e4ccf48d] > > > 4: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, > snapid_t, CInode**, CDentry::linkage_t*)+0x823) [0x55a7e4ccfcb3] > > > 5: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, > snapid_t)+0xbc) [0x55a7e4cd042c] > > > 6: (Locker::_do_cap_update(CInode*, Capability*, int, snapid_t, > boost::intrusive_ptr<MClientCaps const> const&, > boost::intrusive_ptr<MClientCaps> const&, bool*)+0xfb6) [0x55a7e4d957e6] > > > 7: (Locker::handle_client_caps(boost::intrusive_ptr<MClientCaps > const> const&)+0x2059) [0x55a7e4d9c8e9] > > > 8: (Locker::dispatch(boost::intrusive_ptr<Message const> > const&)+0xe7) [0x55a7e4daaf97] > > > 9: (MDSRank::handle_deferrable_message(boost::intrusive_ptr<Message > const> const&)+0x304) [0x55a7e4c089e4] > > > 10: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, > bool)+0x6eb) [0x55a7e4c0b1bb] > > > 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message > const> const&)+0x40) [0x55a7e4c0b8d0] > > > 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> > const&)+0xfc) [0x55a7e4bf82ac] > > > 13: (DispatchQueue::entry()+0x12a9) [0x7f0375968dd9] > > > 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0375a183dd] > > > 15: (()+0x7dd5) [0x7f0373642dd5] > > > 16: (clone()+0x6d) [0x7f03722f302d] > > > > > > 0> 2019-09-19 15:42:00.336 7f036c2f0700 -1 *** Caught signal > (Aborted) ** > > > in thread 7f036c2f0700 thread_name:ms_dispatch > > > > > > ceph version 14.2.3 (0f776cf838a1ae3130b2b73dc26be9c95c6ccc39) > nautilus (stable) > > > 1: (()+0xf5d0) [0x7f037364a5d0] > > > 2: (gsignal()+0x37) [0x7f037222b2c7] > > > 3: (abort()+0x148) [0x7f037222c9b8] > > > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x199) [0x7f0375773b11] > > > 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char > const*, char const*, ...)+0) [0x7f0375773c90] > > > 6: (()+0x1ee48d) [0x55a7e4ccf48d] > > > 7: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, > snapid_t, CInode**, CDentry::linkage_t*)+0x823) [0x55a7e4ccfcb3] > > > 8: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, > snapid_t)+0xbc) [0x55a7e4cd042c] > > > 9: (Locker::_do_cap_update(CInode*, Capability*, int, snapid_t, > boost::intrusive_ptr<MClientCaps const> const&, > boost::intrusive_ptr<MClientCaps> const&, bool*)+0xfb6) [0x55a7e4d957e6] > > > 10: (Locker::handle_client_caps(boost::intrusive_ptr<MClientCaps > const> const&)+0x2059) [0x55a7e4d9c8e9] > > > 11: (Locker::dispatch(boost::intrusive_ptr<Message const> > const&)+0xe7) [0x55a7e4daaf97] > > > 12: (MDSRank::handle_deferrable_message(boost::intrusive_ptr<Message > const> const&)+0x304) [0x55a7e4c089e4] > > > 13: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, > bool)+0x6eb) [0x55a7e4c0b1bb] > > > 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message > const> const&)+0x40) [0x55a7e4c0b8d0] > > > 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> > const&)+0xfc) [0x55a7e4bf82ac] > > > 16: (DispatchQueue::entry()+0x12a9) [0x7f0375968dd9] > > > 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0375a183dd] > > > 18: (()+0x7dd5) [0x7f0373642dd5] > > > 19: (clone()+0x6d) [0x7f03722f302d] > > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > > > > > > > > Thanks! > > > > > > K > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users(a)lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >

1 0

RGW: realm reloader and slow requests
by Eric Choi 19 Sep '19

19 Sep '19

Hello, We recently upgraded from Luminous to Nautilus, after the upgrade, we are seeing this sporadic "lock-up" behavior on the RGW side. What I noticed from the log is that it seems to coincide with rgw realm reloader. What we are seeing is that realm reloader tries to pause frontends, and the last request will take 1 minute or 2 at worst cases, and for that time period RGW is completely locked up, unable to take new requests. 1. Is this an expected behavior? 2. is there a way to disable "rgw realm reloader" or reduce frequency? We are not using multi-site feature and we don't change our realm at all. Thanks!

1 0

unsubscribe
by Oliver Liebel 18 Sep '19

18 Sep '19

1 0

Stability of cephfs snapshot in nautilus
by pinepinebrook 18 Sep '19

18 Sep '19

Is snapshot of cephfs in version nautilus is production ready? When I take a snapshot, what will happen if somebody change the content in that directory before the snapshot finish. Will there be any conflict or something to notice about when I take a snapshot on a busy directory (probably 20 clients r/w in in that directory)

1 0

Ceph deployment tool suggestions
by Shain Miley 17 Sep '19

17 Sep '19

Hello, We currently have an existing 20 node Ceph cluster (17 osd nodes with 3 mon nodes). When this was originally configured, much of the OS install was done manually and the cluster was mainly deployed using ceph-deploy. We are going to be replacing 9 of the 1st gen nodes with 6 newer more dense nodes in the near future. This time around I would like to see about automating the process as much as possible (both OS and Ceph installs). I was wondering if anyone had any suggestions about the best tool to use for this as we are not setting up a cluster from scratch and whatever tool we decide to use, would have to work within the context of an already in production cluster. Some of these tools seem best suited to work when you are setting up a cluster for the very first time, however we are not in that position at this point and I want to make sure I am using something that is flexible enough for our environment going forward. Thanks in advance, Shain -- NPR | Shain Miley | Manager of Infrastructure, Digital Media | smiley(a)npr.org | 202.513.3649

4 3

cephfs and selinux
by Andrey Suharev 17 Sep '19

17 Sep '19

Hi all, I would like to have my home dir at cephfs and to keep selinux enabled at the same time. The trouble is selinux prevents sshd to access ~/.ssh/authorized_keys file. Any ideas how to fix it?

2 2

dashboard not working
by solarflow99 17 Sep '19

17 Sep '19

I have mimic installed and for some reason the dashboard isn't showing up. I see which mon is listed as active for "mgr", the module is enabled, but nothing is listening on port 8080: # ceph mgr module ls { "enabled_modules": [ "dashboard", "iostat", "status" tcp 0 0 10.8.152.4:6800 0.0.0.0:* LISTEN 69049/ceph-mgr

5 8

rados.py with Python3
by tomascribb＠gmail.com 17 Sep '19

17 Sep '19

Hi, i'm integrating ceph like glance's backend, my Openstack is stein over Ubuntu 18.04. When I try upload an image rados.py have an error when calls "run_in_thread()" function (line 238). I've tried use rados mudule from python3 interface and i have the same issue. I read that Python 3 has different behaviour of ctypes c_char_p. How can I resolve that issue?? I copy that example: admin@control1:~$ sudo python3 [sudo] password for admin: Python 3.6.8 (default, Aug 20 2019, 17:12:48) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import rados >>> cluster = rados.Rados(conffile='') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.6/dist-packages/rados.py", line 239, in __init__ (byref(self.cluster), c_char_p(clustername)), TypeError: bytes or integer address expected instead of str instance

1 0

v14.2.4 Nautilus released
by Abhishek Lekshmanan 17 Sep '19

17 Sep '19

This is the fourth release in the Ceph Nautilus stable release series. Its sole purpose is to fix a regression that found its way into the previous release. Notable Changes --------------- The ceph-volume in Nautilus v14.2.3 was found to contain a serious regression, described in https://tracker.ceph.com/issues/41660, which prevented deployment tools like ceph-ansible, DeepSea, Rook, etc. from deploying/removing OSDs. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.3.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 75f4de193b3ea58512f204623e6c5a16e6c1e1ba -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH

1 1

osds xxx have blocked requests > 1048.58 sec / osd.yyy has stuck requests > 67108.9 sec
by Thomas 17 Sep '19

17 Sep '19

Hi, I have defined pool hdd which is exclusively used by virtual disks of multiple KVMs / LXCs. Yesterday I run these commands osdmaptool om --upmap out.txt --upmap-pool hdd source out.txt and Ceph started rebalancing this pool. However since then no KVM / LXC is reacting anymore. If I try to start a new KVM it hangs in boot process. This is the output of ceph health detail: root@ld3955:/mnt/rbd# ceph health detail HEALTH_ERR 28 nearfull osd(s); 1 pool(s) nearfull; Reduced data availability: 1 pg inactive, 1 pg peering; Degraded data redundancy (low space): 8 pgs backfill_toofull; 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio; 2 pools have too many placement groups; 672 slow requests are blocked > 32 sec; 4752 stuck requests are blocked > 4096 sec OSD_NEARFULL 28 nearfull osd(s) osd.42 is near full osd.44 is near full osd.45 is near full osd.77 is near full osd.84 is near full osd.94 is near full osd.101 is near full osd.103 is near full osd.106 is near full osd.109 is near full osd.113 is near full osd.118 is near full osd.120 is near full osd.136 is near full osd.138 is near full osd.142 is near full osd.147 is near full osd.156 is near full osd.159 is near full osd.161 is near full osd.168 is near full osd.192 is near full osd.202 is near full osd.206 is near full osd.208 is near full osd.226 is near full osd.234 is near full osd.247 is near full POOL_NEARFULL 1 pool(s) nearfull pool 'hdb_backup' is nearfull PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg peering pg 30.1b9 is stuck peering for 4722.750977, current state peering, last acting [183,27,63] PG_DEGRADED_FULL Degraded data redundancy (low space): 8 pgs backfill_toofull pg 11.465 is active+remapped+backfill_wait+backfill_toofull, acting [308,351,58] pg 11.5c4 is active+remapped+backfill_wait+backfill_toofull, acting [318,336,54] pg 11.afd is active+remapped+backfill_wait+backfill_toofull, acting [347,220,315] pg 11.b82 is active+remapped+backfill_toofull, acting [314,320,254] pg 11.1803 is active+remapped+backfill_wait+backfill_toofull, acting [88,363,302] pg 11.1aac is active+remapped+backfill_wait+backfill_toofull, acting [328,275,95] pg 11.1c09 is active+remapped+backfill_wait+backfill_toofull, acting [55,124,278] pg 11.1e36 is active+remapped+backfill_wait+backfill_toofull, acting [351,92,315] POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['hdb_backup'] overcommit available storage by 1.708x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['hdb_backup'] overcommit available storage by 1.708x due to target_size_ratio 0.000 on pools [] POOL_TOO_MANY_PGS 2 pools have too many placement groups Pool hdd has 512 placement groups, should have 128 Pool pve_cephfs_metadata has 32 placement groups, should have 4 REQUEST_SLOW 672 slow requests are blocked > 32 sec 249 ops are blocked > 2097.15 sec 284 ops are blocked > 1048.58 sec 108 ops are blocked > 524.288 sec 9 ops are blocked > 262.144 sec 22 ops are blocked > 131.072 sec osd.9 has blocked requests > 524.288 sec osds 0,2,6,68 have blocked requests > 1048.58 sec osd.3 has blocked requests > 2097.15 sec REQUEST_STUCK 4752 stuck requests are blocked > 4096 sec 1431 ops are blocked > 67108.9 sec 513 ops are blocked > 33554.4 sec 909 ops are blocked > 16777.2 sec 1809 ops are blocked > 8388.61 sec 90 ops are blocked > 4194.3 sec osd.63 has stuck requests > 67108.9 sec My interpretation is that Ceph a) is busy with remapping PGs of pool hdb_backup b) has identified several OSDs with either blocked or stuck requests. Any of these OSDs belongs to pool hdd, though. osd.9 belongs to node A, osd.63 and osd.68 belongs to node C (there are 4 nodes serving OSD in the cluster). I have tried to fix this issue, but it failed with - ceph osd set noout - restart of relevant OSD by systemctl restart ceph-osd@<id> and finally server reboot. I also tried to migrate the virtual disks to another pool, but this fails, too. There are no changes on server side, like network or disks or whatsoever. How can I resolve this issue? THX Thomas

2 1

upmap supported in SLES 12SPx
by Thomas 17 Sep '19

17 Sep '19

Hi, I tried to run this command with failure: root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous Error EPERM: cannot set require_min_compat_client to luminous: 6 connected client(s) look like jewel (missing 0xa00000000200000); 19 connected client(s) look like jewel (missing 0x800000000000000); 23 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway root@ld3955:/mnt/rbd# ceph features { "mon": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ], "mds": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 2 } ], "osd": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 368 } ], "client": [ { "features": "0x40106b84a842a42", "release": "jewel", "num": 6 }, { "features": "0x27018eb84aa42a52", "release": "jewel", "num": 19 }, { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 23 }, { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 35 } ], "mgr": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ] } But I think the number of clients older than Luminous is incorrect. I'm running 98% of the clients with SLES 12SPx, and therefore the question is: Can you confirm in which SLES 12 release the function upmap is supported? And is it safe to execute ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it? What happens to the clients then? Regards Thomas

8 14

Re: download.ceph.com repository changes
by vitalif＠yourcmc.ru 17 Sep '19

17 Sep '19

The worst part about the official repository is that it lacks Debian packages Also of course it would be very convenient to be able to install any version from the repos, not just the latest one. It's certainly possible with debian repos...

1 0

Unsubscribe
by Rimma Iontel 17 Sep '19

17 Sep '19

2 1

14.2.4 Packages Avaliable
by Ashley Merrick 17 Sep '19

17 Sep '19

Have just noticed their is packages available for 14.2.4.. I know with the whole 14.2.3 release and the notes not going out to a good day or so later.. but this is not long after the 14.2.3 release..? Was this release even meant to have come out? Makes it difficult for people installing a new node if they can't reply on the current "stable" packages that apt/yum will give them.

10 12

New lines "choose_args" in crush map
by Thomas 17 Sep '19

17 Sep '19

Hi! I recently noticed new lines in my crush map. Here's an extract of it: # choose_args choose_args 18446744073709551615 { { bucket_id -1 weight_set [ [ 33.839 34.159 34.041 34.768 1.810 1.832 ] ] } { bucket_id -2 weight_set [ [ 28.019 28.339 28.221 28.948 1.810 1.832 ] ] } { bucket_id -3 weight_set [ [ 2.910 2.910 1.663 1.605 1.646 1.646 1.663 1.694 1.671 1.663 1.605 1.678 1.617 1.656 1.678 1.588 1.653 1.635 1.660 ] ] } { bucket_id -4 weight_set [ [ 1.663 1.605 1.646 1.646 1.663 1.694 1.671 1.663 1.605 1.678 1.617 1.656 1.678 1.588 1.653 1.635 1.660 ] ] } { bucket_id -5 weight_set [ [ 2.910 2.910 ] ] } { bucket_id -6 weight_set [ [ 5.820 5.820 5.820 5.820 0.000 0.000 ] ] } [...] Question: What is writing these lines into crush map? What is the function behind these lines? THX Thomas

1 0

KRBD use Luminous upmap feature.Which version of the kernel should i ues?
by 潘东元 17 Sep '19

17 Sep '19

hi, my ceph cluster version is Luminous run the kernel version Linux 3.10 [root@node-1 ~]# ceph features { "mon": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 3 } }, "osd": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 15 } }, "client": { "group": { "features": "0x40106b84a842a52", "release": "jewel", "num": 3 }, "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 170 } } } Which version of the kernel should i ues? Thanks!

3 5

Nautilus: pg_autoscaler causes mon slow ops
by Eugen Block 17 Sep '19

17 Sep '19

Hi everyone, we upgraded our production cluster just recently to version: ceph version 14.2.3-349-g7b1552ea82 (7b1552ea827cf5167b6edbba96dd1c4a9dc16937) nautilus (stable) We then activated pg_autoscaler for two pools that had a bad pg_num and the result is satisfying. However, after the rebalance finished the cluster became laggy. We noticed that two out of three MONs had a much higher CPU usage than usual, according to `top` the MON processes consumed more than 100%. Restarting the MON services and disabling pg_autoscaler resolved the issue. I've read that the balancer module can cause a higher load on the MGR daemon, is this somehow related? Another thing to mention is the confusing calculation of the autoscaler. After the pg numbers had been corrected we got the warning about overcommitted pools: > 1 subtrees have overcommitted pool target_size_bytes > 1 subtrees have overcommitted pool target_size_ratio The images pool was responsible for that. The confusing part was that sometimes autoscale-status displayed the size of that pool with more than 14 TB: ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE images 14399G 3.0 33713G 1.2813 1.0 128 on And a couple of minutes later the pool suddenly only had around 4 TB of data: ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE images 4112G 3.0 33713G 0.3659 1.0 128 on There seems to be some kind of inconsistency here. The actual used storage of this pool according to `ceph df` is: POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL images 1 4.1 TiB 1.01M 12 TiB 49.73 4.1 TiB Has anyone experienced something similar? Are these known issues? Regards, Eugen

1 0

Intergrate Metadata with ElasticSeach
by tuan dung 17 Sep '19

17 Sep '19

Dear all, Can you show me steps how to intergrate Metadata of Ceph object with ElasticSeach to improve searching medata performance? thank you very much. ----------------------------- Br, Dương Tuấn Dũng 0986153686

1 0

Fwd: ceph-users Digest, Vol 80, Issue 54
by Rom Freiman 16 Sep '19

16 Sep '19

unsubscribe ---------- Forwarded message --------- From: <ceph-users-request(a)ceph.io> Date: Mon, Sep 16, 2019 at 7:22 PM Subject: ceph-users Digest, Vol 80, Issue 54 To: <ceph-users(a)ceph.io> Send ceph-users mailing list submissions to ceph-users(a)ceph.io To subscribe or unsubscribe via email, send a message with subject or body 'help' to ceph-users-request(a)ceph.io You can reach the person managing the list at ceph-users-owner(a)ceph.io When replying, please edit your Subject line so it is more specific than "Re: Contents of ceph-users digest..." Today's Topics: 1. Re: upmap supported in SLES 12SPx (Ilya Dryomov) 2. Re: upmap supported in SLES 12SPx (Thomas Schneider) 3. Re: upmap supported in SLES 12SPx (Ilya Dryomov) 4. Re: Using same instance name for rgw (Eric Choi) 5. Re: RGW Passthrough (Casey Bodley) ---------------------------------------------------------------------- Date: Mon, 16 Sep 2019 16:56:19 +0200 From: Ilya Dryomov <idryomov(a)gmail.com> Subject: [ceph-users] Re: upmap supported in SLES 12SPx To: Thomas Schneider <74cmonty(a)gmail.com> Cc: Konstantin Shalygin <k0ste(a)k0ste.ru>, ceph-users <ceph-users(a)ceph.io> Message-ID: <CAOi1vP-YzuMeL-u=hPDMr6fSTWd+F7RXX61kfO9609pxv98qNw(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" On Mon, Sep 16, 2019 at 4:40 PM Thomas Schneider <74cmonty(a)gmail.com> wrote: > > Hi, > > thanks for your valuable input. > > Question: > Can I get more information of the 6 clients (those with features > 0x40106b84a842a42), e.g. IP, that allows me to identify it easily? Yes, although it's not integrated into "ceph features". Log into a monitor node and run "ceph daemon mon.a sessions" (mon.a is the name of the monitor, substitute accordingly). Thanks, Ilya ------------------------------ Date: Mon, 16 Sep 2019 17:10:37 +0200 From: Thomas Schneider <74cmonty(a)gmail.com> Subject: [ceph-users] Re: upmap supported in SLES 12SPx To: Ilya Dryomov <idryomov(a)gmail.com> Cc: Konstantin Shalygin <k0ste(a)k0ste.ru>, ceph-users <ceph-users(a)ceph.io> Message-ID: <79b342ac-be00-e3e7-cbec-b6c96d3a0a59(a)gmail.com> Content-Type: text/plain; charset=utf-8 Wonderbra. I found some relevant sessions on 2 of 3 monitor nodes. And I found some others: root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42 root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous [ "MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52 (jewel))", ] Would it make sense to shutdown these clients, too? What confuses me is that the list includes clients that belong to the Ceph cluster, namely 10.97.206.0/24. All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph. Regards Thomas Am 16.09.2019 um 16:56 schrieb Ilya Dryomov: > On Mon, Sep 16, 2019 at 4:40 PM Thomas Schneider <74cmonty(a)gmail.com> wrote: >> Hi, >> >> thanks for your valuable input. >> >> Question: >> Can I get more information of the 6 clients (those with features >> 0x40106b84a842a42), e.g. IP, that allows me to identify it easily? > Yes, although it's not integrated into "ceph features". Log into > a monitor node and run "ceph daemon mon.a sessions" (mon.a is the name > of the monitor, substitute accordingly). > > Thanks, > > Ilya ------------------------------ Date: Mon, 16 Sep 2019 17:36:35 +0200 From: Ilya Dryomov <idryomov(a)gmail.com> Subject: [ceph-users] Re: upmap supported in SLES 12SPx To: Thomas Schneider <74cmonty(a)gmail.com> Cc: Konstantin Shalygin <k0ste(a)k0ste.ru>, ceph-users <ceph-users(a)ceph.io> Message-ID: <CAOi1vP9MsfoQFPNfUYPRn2MzS=5buZ-s2Jcorkym84K8hJ6cWw(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" On Mon, Sep 16, 2019 at 5:10 PM Thomas Schneider <74cmonty(a)gmail.com> wrote: > > Wonderbra. > > I found some relevant sessions on 2 of 3 monitor nodes. > And I found some others: > root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42 > root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous > [ > "MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open > allow *, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open > allow *, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open > allow r, features 0x27018eb84aa42a52 (jewel))", > "MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open > allow r, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open > allow r, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open > allow *, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open > allow r, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow > r, features 0x27018eb84aa42a52 (jewel))", > "MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open > allow r, features 0x27018eb84aa42a52 (jewel))", > "MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open > allow r, features 0x27018eb84aa42a52 (jewel))", > "MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open > allow r, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow > r, features 0x27018fb86aa42ada (jewel))", > "MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52 > (jewel))", > ] > > Would it make sense to shutdown these clients, too? > > What confuses me is that the list includes clients that belong to the > Ceph cluster, namely 10.97.206.0/24. > All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph. The above output seems consistent with your "ceph features" output: it lists clients with features 0x27018eb84aa42a52 and 0x27018fb86aa42ada. Like I said in my previous email, both of these support upmap. If you temporarily shut them down, set-require-min-compat-client will work without --yes-i-really-mean-it. Thanks, Ilya ------------------------------ Date: Mon, 16 Sep 2019 16:05:47 -0000 From: "Eric Choi" <eric.yongjun.choi(a)gmail.com> Subject: [ceph-users] Re: Using same instance name for rgw To: ceph-users(a)ceph.io Message-ID: <156864994756.18.7543223789861447910@mailman-web> Content-Type: text/plain; charset="utf-8" bump. anyone? ------------------------------ Date: Mon, 16 Sep 2019 12:22:16 -0400 From: Casey Bodley <cbodley(a)redhat.com> Subject: [ceph-users] Re: RGW Passthrough To: ceph-users(a)ceph.io Message-ID: <fdf93930-794e-b2d0-46ee-5288a4d91605(a)redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Hi Robert, So far the cloud tiering features are still in the design stages. We're working on some initial refactoring work to support this abstraction (ie. to either satisfy a request against the local rados cluster, or to proxy it somewhere else). With respect to passthrough/tiering to AWS, we could use help thinking through the user/credential mapping in particular. We have a weekly 'RGW Refactoring' meeting on Wednesdays where we discuss design and refactoring progress - it's on the upstream community calendar, I'll send you an invite. On 9/13/19 9:59 PM, Robert LeBlanc wrote: > We are very interested in the RGW Passthrough mentioned for Octupus. > What's the status and how can we help? We want to connect with AWS S3. > > Thank you, > ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------ Subject: Digest Footer _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s ------------------------------ End of ceph-users Digest, Vol 80, Issue 54 ******************************************

1 0

RGW Passthrough
by Robert LeBlanc 16 Sep '19

16 Sep '19

We are very interested in the RGW Passthrough mentioned for Octupus. What's the status and how can we help? We want to connect with AWS S3. Thank you, ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 1

Using same instance name for rgw
by Eric Choi 16 Sep '19

16 Sep '19

I previously posted this question to lists.ceph.com not understanding lists.ceph.io is the replacement for it. Posting it again here with some edits. --- Hi there, we have been using ceph for a few years now, it's only now that I've noticed we have been using the same name for all RGW hosts, resulting when you run ceph -s: rgw: 1 daemon active (..) also our ceph.conf looks like (for rgw) ... [client.radosgw.gateway] ... despite having more than 10 RGW hosts. * What are the side effects of doing this? Is this a no-no? I can see the metrics can (ceph daemon ... perf dump) be wrong, are the metrics kept track independently (per host)? Can this affect performance negatively by any chance? (we are using the same key obviously..) * We recently upgraded from Lumunious to Nautilus, I've noticed that later docs all prescribes radosgw config section as [client.rgw.{instance-name}] .. should we make this change? Much appreciated!

1 1

Different pools count in ceph -s and ceph osd pool ls
by Fyodor Ustinov 16 Sep '19

16 Sep '19

Hi! I create bug https://tracker.ceph.com/issues/41832 Maybe someone also encountered such a problem? WBR, Fyodor.

1 0

Re: Ceph Day London - October 24 (Call for Papers!)
by Wido den Hollander 16 Sep '19

16 Sep '19

Hi, The CFP is ending today for the Ceph Day London on October 24th. If you have a talk you would like to submit, please follow the link below! Wido On 7/18/19 3:43 PM, Wido den Hollander wrote: > Hi, > > We will be having Ceph Day London October 24th! > > https://ceph.com/cephdays/ceph-day-london-2019/ > > The CFP is now open for you to get your Ceph related content in front > of the Ceph community ranging from all levels of expertise: > > https://forms.zohopublic.com/thingee/form/CephDayLondon2019/formperma/h96jZ… > > If your company is interested in sponsoring the event, we would be > delighted to have you. Please contact me directly for further information. > > The Ceph Day is co-located with the Apache CloudStack project. There > will be two tracks where people can choose between Ceph and CloudStack. > > After the Ceph Day there's going to be beers in the pub nearby to make > new friends. > > Join us in London on October 24th! > > Wido > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

1 0

Warning: 1 pool nearfull and unbalanced data distribution
by Thomas 16 Sep '19

16 Sep '19

Update: I have been pointed to this <https://docs.ceph.com/docs/master/rados/operations/upmap/> doku "Using the pg-upmap" and I checked the offline optimization. However the output for the relevant pool shows: no upmaps proposed root@ld3955:/mnt/rbd# osdmaptool om --upmap out.txt --upmap-pool hdb_backup osdmaptool: osdmap file 'om' writing upmap command output to: out.txt checking for upmap cleanups upmap, max-count 100, max deviation 0.01 limiting to pools hdb_backup (11) no upmaps proposed This is very strange because it is clear that this pool hdb_backup is very much unbalanced. Please advise. THX .......................................... Hi, the output of ceph health details gives me a warning that concerns me a little. I'll explain in a second. root@ld3955:/mnt/rbd# ceph health detail HEALTH_WARN 1 nearfull osd(s); 1 pool(s) nearfull; 4 pools have too many placement groups OSD_NEARFULL 1 nearfull osd(s) osd.122 is near full POOL_NEARFULL 1 pool(s) nearfull pool 'hdb_backup' is nearfull POOL_TOO_MANY_PGS 4 pools have too many placement groups Pool pve_cephfs_data has 128 placement groups, should have 16 Pool hdd has 512 placement groups, should have 64 Pool pve_cephfs_metadata has 32 placement groups, should have 4 Pool backup has 1024 placement groups, should have 4 I'm writing +90% of the data in pool "hdb_backup", and this is ongoing. Therefore I can hardly afford that this pool is full. When I check the output of ceph osd status the relevand osd is somehow overutilized: root@ld3955:/mnt/rbd# ceph osd status | grep nearfull +-----+--------+-------+-------+--------+---------+--------+---------+--------------------+ | id | host | used | avail | wr ops | wr data | rd ops | rd data | state | +-----+--------+-------+-------+--------+---------+--------+---------+--------------------+ | 122 | ld5505 | 1448G | 227G | 0 | 0 | 0 | 0 | exists,nearfull,up | This looks like an inconsistency, but I can check the usage with ceph osd df. Here I can see that the OSDs are not really balanced: root@ld3955:/mnt/rbd# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 272 hdd 7.28000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 136 KiB 3.7 GiB 5.0 TiB 31.23 0.77 121 up 273 hdd 7.28000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 28 KiB 5.1 GiB 4.4 TiB 39.35 0.97 152 up 274 hdd 7.28000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 168 KiB 4.5 GiB 4.4 TiB 39.41 0.97 152 up 275 hdd 7.28000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 139 KiB 3.5 GiB 5.1 TiB 29.77 0.73 115 up 276 hdd 7.28000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 48 KiB 5.7 GiB 4.5 TiB 38.81 0.96 150 up 277 hdd 7.28000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 276 KiB 4.3 GiB 4.8 TiB 34.68 0.85 134 up 278 hdd 7.28000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 36 KiB 4.4 GiB 4.5 TiB 38.74 0.95 150 up 279 hdd 7.28000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 156 KiB 4.1 GiB 4.7 TiB 35.80 0.88 138 up 280 hdd 7.28000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 156 KiB 4.3 GiB 4.6 TiB 37.03 0.91 143 up 281 hdd 7.28000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 172 KiB 3.8 GiB 4.9 TiB 32.67 0.80 126 up 282 hdd 7.28000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 120 KiB 4.5 GiB 4.4 TiB 39.39 0.97 152 up 283 hdd 7.28000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 32 KiB 5.9 GiB 4.5 TiB 37.57 0.93 145 up [...] 76 hdd 1.64000 1.00000 1.6 TiB 1.4 TiB 1.4 TiB 88 KiB 2.4 GiB 268 GiB 84.02 2.07 73 up 77 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 188 KiB 2.0 GiB 560 GiB 66.59 1.64 58 up 78 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1023 GiB 164 KiB 1.9 GiB 651 GiB 61.15 1.51 53 up 79 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 176 KiB 1.9 GiB 636 GiB 62.02 1.53 54 up 80 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 80 KiB 2.5 GiB 636 GiB 62.07 1.53 54 up 81 hdd 1.64000 1.00000 1.6 TiB 886 GiB 885 GiB 128 KiB 1.7 GiB 790 GiB 52.89 1.30 46 up 82 hdd 1.64000 1.00000 1.6 TiB 967 GiB 965 GiB 240 KiB 1.8 GiB 709 GiB 57.70 1.42 50 up 83 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.2 GiB 420 GiB 74.94 1.85 65 up 84 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 108 KiB 2.0 GiB 597 GiB 64.37 1.59 56 up 85 hdd 1.64000 1.00000 1.6 TiB 811 GiB 810 GiB 176 KiB 1.6 GiB 865 GiB 48.42 1.19 42 up 86 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 72 KiB 2.0 GiB 613 GiB 63.43 1.56 55 up 87 hdd 1.64000 1.00000 1.6 TiB 791 GiB 789 GiB 68 KiB 1.6 GiB 885 GiB 47.17 1.16 41 up 88 hdd 1.64000 1.00000 1.6 TiB 908 GiB 906 GiB 168 KiB 1.8 GiB 768 GiB 54.18 1.33 47 up [...] 113 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 100 KiB 3.0 GiB 342 GiB 79.60 1.96 69 up 114 hdd 1.64000 1.00000 1.6 TiB 1001 GiB 999 GiB 184 KiB 1.9 GiB 675 GiB 59.70 1.47 52 up 115 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.2 GiB 407 GiB 75.70 1.86 66 up 116 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 92 KiB 2.0 GiB 597 GiB 64.39 1.59 56 up 117 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 76 KiB 2.7 GiB 480 GiB 71.34 1.76 62 up 118 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 48 KiB 2.6 GiB 574 GiB 65.74 1.62 57 up 119 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 152 KiB 1.9 GiB 634 GiB 62.19 1.53 54 up 120 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 48 KiB 2.0 GiB 541 GiB 67.73 1.67 59 up 121 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 48 KiB 2.0 GiB 556 GiB 66.82 1.65 58 up 122 hdd 1.64000 0.95001 1.6 TiB 1.4 TiB 1.4 TiB 184 KiB 2.5 GiB 227 GiB 86.44 2.13 75 up I assume that this is a result of my Ceph Cluster history, means I started with 4 OSD nodes with 48 drives @1.8TB. This was 345TB in total. I started to fill this storage up to 75%. Then I added 2 OSD nodes with 48 drives @8TB. I was not expecting that pool "hdb_backup" would be filled up in the near future. This is the ouput of ceph df: root@ld3955:/mnt/rbd# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.1 PiB 661 TiB 467 TiB 468 TiB 41.43 nvme 23 TiB 23 TiB 681 MiB 8.7 GiB 0.04 TOTAL 1.1 PiB 685 TiB 467 TiB 468 TiB 40.59 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL backup 4 0 B 0 0 B 0 35 TiB hdb_backup 11 154 TiB 40.39M 154 TiB 64.18 29 TiB hdd 30 1.1 TiB 281.21k 1.1 TiB 1.01 35 TiB pve_cephfs_data 32 318 GiB 91.83k 318 GiB 0.30 35 TiB pve_cephfs_metadata 33 155 MiB 61 155 MiB 0 35 TiB nvme 35 0 B 0 0 B 0 7.4 TiB Question: How can I start rebalancing data in order to have more data in the larger drives (8TB)? Or is it ok that the smaller drives (1.8TB) are filled by +60%? THX for your advice Thomas

1 0

Delete objects on large bucket very slow
by tuan dung 13 Sep '19

13 Sep '19

Dear Ceph team, I builde a object storage using ceph, a cluster with 5 nodes (9 sas10k 2TB (for data)+ 1 nvme (for metadata) in one node , using ceph version 14.2.3, nautilus (stable). My cluster storage over 100M (minlion) objects (files) with file's size ~ 50KB, and I want to delete them to free capacity and improve performance. When I delete object, speed delete object very slow, about ~ 35-36 objs/s at 100M (minlion) objects point, i using command: s3cmd del -r s3://mybucket/2019 to delete. Can you help me: - How does ceph's deleting object operator work? (it mean about how delete object flow in ceph works?) - how to improve speed of delete object operator ( objects/s) - how to delete mult - bestpractice for this usecase - other recommend.... Thank you very much. ----------------------------- Br, Dương Tuấn Dũng 0986153686

3 4

Legacy Bluestore Stats
by gaving＠voxtandem.com 12 Sep '19

12 Sep '19

I created a cluster on Nautilus 14.2.0, then upgraded to 14.2.1, and finally 14.2.3 Now I am seeing this warning that I thought should only appear if the cluster was created pre-Nautilus. Legacy BlueStore stats reporting detected on XX OSD I can't seem to find any information about this on a cluster that was always on 14.2 Gavin

1 0

Bug identified: Dashboard proxy configuration is not working as expected
by Thomas 12 Sep '19

12 Sep '19

Hi, I have successfully configured Ceph dashboard following the this <https://docs.ceph.com/docs/master/mgr/dashboard> documentation. According to the documentation you can configure a URL prefix with this command: ceph config set mgr mgr/dashboard/url_prefix $PREFIX However when I try to access the dashboard with this URL http://$IP:$PORT/$PREFIX/ I get an error: {"status": "404 Not Found", "version": "8.9.1", "detail": "The path '/dashboard' was not found.", "traceback": "Traceback (most recent call last):\n File \"/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py\", line 670, in respond\n response.body = self.handler()\n File \"/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py\", line 220, in __call__\n self.body = self.oldhandler(*args, **kwargs)\n File \"/usr/lib/python2.7/dist-packages/cherrypy/_cperror.py\", line 415, in __call__\n raise self\nNotFound: (404, \"The path '/dashboard' was not found.\")\n"} A workaround for this error is to use this URL: http://$IP:$PORT/#/$PREFIX/ Using the # ensures that redirect to active MGR service is working. Regards Thomas

2 1

645% Clean PG's in Dashboard
by c.lilja＠falseprivacy.org 12 Sep '19

12 Sep '19

Hi, I've upgraded to Nautilus from Mimic a while ago and enabled the pg_autoscaler. When pg_autoscaler was activated I got a HEALTH_WARN regarding: POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data_reduced', 'cephfs_data', 'cephfs_metadata'] overcommit available storage by 1.460x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data_reduced', 'cephfs_data', 'cephfs_metadata'] overcommit available storage by 1.460x due to target_size_ratio 0.000 on pools [] Both target_size_bytes and target_size_ratio on all the pools are set to 0, so I started to wonder why this error message appear. My autoscale-status looks like this: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 16708M 4.0 34465G 0.0019 1.0 8 warn cephfs_data_reduced 15506G 2.0 34465G 0.8998 1.0 375 warn cephfs_data 6451G 3.0 34465G 0.5616 1.0 250 warn So the ratio in total is 1.4633.. Isn't 1.0 of the combined ratio of all pools equal of full? I also enabled the Dashboard and saw that the PG Status showed "645% clean" PG's. This cluster was originally installed with version Jewel, so may it be any legacy setting or such that causing this?

2 1

the different between flag system and admin when create user rgw
by Wahyu Muqsita 12 Sep '19

12 Sep '19

when create user rgw using command : radosgw-admin user create --uid={username} --display-name="{display-name}" [--email={email}] There are 2 flag that I can use —system and —admin. What is this flag for ? -- ------------------------------------------------------------ Wahyu Muqsita Wardana System Engineer ------------------------------------------------------------ Jl. Ampera Raya Nomor 22, Cilandak Timur Jakarta Selatan 12560, Indonesia. T. +62217182008 | M. +62 8227 3185 744 www.bukalapak.com This message and any attachments are confidential and intended solely for the use of the individual to whom it is addressed.

2 1

How to use radosgw-min find ?
by EDH - Manuel Rios Fernandez 12 Sep '19

12 Sep '19

Hi! We're looking to mantain our rgw pools out of orphans objects, checking the documentation and mailist is not really clear how it works and what will do. Radosgw-admin orphands find -pool= --job-id= Loops over all objects in the cluster looking for leaked objects and add it to a shard in the pool rgw.log. For us after more than 72 hours running stuck with 24 GB ram used Console show : 7fc6ca719700 0 run(): building index of all bucket indexes 7fc6ca719700 0 run(): building index of all linked objects 7fc6ca719700 0 building linked oids index: 0/64 7fc6ca719700 0 building linked oids index: 1/64 Checking the RGW.log pool it generated 64 largeomaps log file. Anyone got experience with orphans objects? We calculated near 80-100TB orphans objects in our cluster. Regards Manuel

1 0

CentOS deps for ceph-mgr-diskprediction-local
by Dan van der Ster 12 Sep '19

12 Sep '19

Hi there, Did anyone get the mgr diskprediction-local plugin working on CentOS ? When I enable the plugin with v14.2.3 I get: HEALTH_ERR 2 mgr modules have failed MGR_MODULE_ERROR 2 mgr modules have failed Module 'devicehealth' has failed: Failed to import _strptime because the import lockis held by another thread. Module 'diskprediction_local' has failed: No module named sklearn.svm.classes When the package is installed it brings in several deps but apparently these are not enough? Installing: ceph-mgr-diskprediction-local noarch 2:14.2.3-0.el7 ceph-noarch 1.1 M Installing for dependencies: atlas x86_64 3.10.1-12.el7 base 4.5 M blas x86_64 3.4.2-8.el7 base 399 k lapack x86_64 3.4.2-8.el7 base 5.4 M libgfortran x86_64 4.8.5-39.el7 cr 300 k libquadmath x86_64 4.8.5-39.el7 cr 190 k numpy x86_64 1:1.7.1-13.el7 base 2.8 M numpy-f2py x86_64 1:1.7.1-13.el7 base 206 k python-devel x86_64 2.7.5-86.el7 cr 398 k python-nose noarch 1.3.7-1.el7 base 276 k python-rpm-macros noarch 3-32.el7 cr 8.8 k python-srpm-macros noarch 3-32.el7 cr 8.4 k python2-rpm-macros noarch 3-32.el7 cr 7.7 k scipy x86_64 0.12.1-6.el7 base 9.3 M suitesparse x86_64 4.0.2-10.el7 base 928 k tbb x86_64 4.1-9.20130314.el7 base 124 k I've seen https://tracker.ceph.com/issues/38088 but didn't find the sklearn package in any standard repo. Thanks! Dan

1 0

MDSs report slow metadata IOs
by burcarjo＠gmail.com 12 Sep '19

12 Sep '19

I have a cephFS 13.2.6 setup composed by 3 OSDs nodes + 1 MDS + 1 monitor. All the nodes are working with CentOS Linux release 7.6.1810 (Core) When writing multiple lot of files (500MB) then the "ls" command of this directory works very slow from an external client (if I list the directory from the same client node that is writing the files then the operation returns immediately): [cephuser@stor2demo ~]$ time ll -h /mnt/cephfs/dir1/dir2 real 2m40.246s user 0m0.002s sys 0m0.003s The Ceph logs shows this information: [cephuser@stor1demo ~]$ ceph health detail HEALTH_WARN 1 MDSs report slow metadata IOs; 1 MDSs report slow requests; 1/10269 objects misplaced (0.010%) MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs mdsstor1demo(mds.0): 5 slow metadata IOs are blocked > 30 secs, oldest blocked for 34 secs MDS_SLOW_REQUEST 1 MDSs report slow requests mdsstor1demo(mds.0): 2 slow requests are blocked > 30 secs OBJECT_MISPLACED 1/10269 objects misplaced (0.010%) Why this behaviour ? Thanks.

2 2

Re: ceph-volume lvm create leaves half-built OSDs lying around
by Jan Fajerski 12 Sep '19

12 Sep '19

On Wed, Sep 11, 2019 at 11:17:47AM +0100, Matthew Vernon wrote: >Hi, > >We keep finding part-made OSDs (they appear not attached to any host, >and down and out; but still counting towards the number of OSDs); we >never saw this with ceph-disk. On investigation, this is because >ceph-volume lvm create makes the OSD (ID and auth at least) too early in >the process and is then unable to roll-back cleanly (because the >bootstrap-osd credential isn't allowed to remove OSDs). > >As an example (very truncated): > >Running command: /usr/bin/ceph --cluster ceph --name >client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring >-i - osd new 20cea174-4c1b-4330-ad33-505a03156c33 >Running command: vgcreate --force --yes >ceph-9d66ec60-c71b-49e0-8c1a-e74e98eafb0e /dev/sdbh > stderr: Device /dev/sdbh not found (or ignored by filtering). > Unable to add physical volume '/dev/sdbh' to volume group >'ceph-9d66ec60-c71b-49e0-8c1a-e74e98eafb0e'. >--> Was unable to complete a new OSD, will rollback changes >--> OSD will be fully purged from the cluster, because the ID was generated >Running command: ceph osd purge osd.828 --yes-i-really-mean-it > stderr: 2019-09-10 15:07:53.396528 7fbca2caf700 -1 auth: unable to find >a keyring on >/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >(2) No such file or directory > stderr: 2019-09-10 15:07:53.397318 7fbca2caf700 -1 monclient: >authenticate NOTE: no keyring found; disabled cephx authentication >2019-09-10 15:07:53.397334 7fbca2caf700 0 librados: client.admin >authentication error (95) Operation not supported > >This is annoying to have to clear up, and it seems to me could be >avoided by either: > >i) ceph-volume should (attempt to) set up the LVM volumes &c before >making the new OSD id >or >ii) allow the bootstrap-osd credential to purge OSDs > >i) seems like clearly the better answer...? Agreed. Would you mind opening a bug report on https://tracker.ceph.com/projects/ceph-volume. I have found other situation where a roll-back is working as it should, though not with as much impact as this. > >Regards, > >Matthew > >_______________________________________________ >ceph-users mailing list >ceph-users(a)lists.ceph.com >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH (HRB 247165, AG München) Geschäftsführer: Felix Imendörffer

2 1

subscriptions from lists.ceph.com now on lists.ceph.io?
by Matthias Ferdinand 11 Sep '19

11 Sep '19

Hi, sorry to disturb you with list admin stuff. I haven't received any new ceph-users@ mail after August, 28th (I was subscribed to daily digest). lists.ceph.com now is defunct, mails are bounced. I hope that is only temporary, because most of my annotated ceph bookmarks point to lists.ceph.com... From the website it looks like the list was moved to ceph.io. I don't remember reading any announcement of this move (might be my fault). Tried re-subscribing at lists.ceph.io, but it says I am already subscribed. Tried logging in to check preferences, but my old password from lists.ceph.com does not work anymore. Created a new account, logged in and to me the subscription settings look ok. Can you help me here? Maybe it is just the digests that do not work? Please answer to me directly, as I am currently not receiving any list messages. Thank you Matthias Ferdinand

2 1

Re: regurlary 'no space left on device' when deleting on cephfs
by Burkhard Linke 11 Sep '19

11 Sep '19

Hi, do you use hard links in your workload? The 'no space left on device' message may also refer to too many stray files. Strays are either files that are to be deleted (e.g. the purge queue), but also files which are deleted, but hard links are still pointing to the same content. Since cephfs does not use an indirect layer between inodes and data, and the data chunks are named after the inode id, removing the original file will leave stray entries since cephfs is not able to rename the underlying rados objects. There are 10 hidden directories for stray files, and given a maximum size of 100.000 entries you can store only up to 1 million entries. I don't know exactly how entries are distributed among the 10 directories, so the limit may be reached earlier for a single stray directory. The performance counters contains some values for stray, so they are easy to check. The daemonperf output also shows the current value. The problem of the upper limit of directory entries was solved by directory fragmentation, so you should check whether fragmentation is allowed in your filesystem. You can also try to increase the upper directory entry limit, but this might lead to other problems (too large rados omap objects....). Regards, Burkhard -- Dr. rer. nat. Burkhard Linke Bioinformatics and Systems Biology Justus-Liebig-University Giessen 35392 Giessen, Germany Phone: (+49) (0)641 9935810

3 3