- ceph-users - lists.ceph.io

Activate Cache Tier on Running Pools
by Eikermann, Robert 23 Sep '19

23 Sep '19

Hi, I'm using Ceph in combination with Openstack. For the "VMs" Pool I'd like to enable writeback caching tier, like described here: https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ . Should it be possible to do that on a running pool? I tried to do so and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly filesystems. No errors were shown in ceph (but also no traffic arrived after enabling the cache tier). Removing the cache tier , rebooting the VMs and doing a filesystemcheck repaired everything. Best Robert

6 9

Seemingly unbounded osd_snap keys in monstore. Normal? Expected?
by Koebbe, Brian 23 Sep '19

23 Sep '19

Our cluster has a little over 100 RBDs. Each RBD is snapshotted with a typical "frequently", hourly, daily, monthly type of schedule. A while back a 4th monitor was temporarily added to the cluster that took hours to synchronize with the other 3. While trying to figure out why that addition took so long, we discovered that our monitors have what seems like a really large number of osd_snap keys: ceph-monstore-tool /var/lib/ceph/mon/xxxxxx dump-keys |awk '{print $1}'|uniq -c 153 auth 2 config 10 health 1441 logm 3 mdsmap 313 mgr 1 mgr_command_descs 3 mgr_metadata 163 mgrstat 1 mkfs 323 mon_config_key 1 mon_sync 6 monitor 1 monitor_store 32 monmap 120 osd_metadata 1 osd_pg_creating 5818618 osd_snap 41338 osdmap 754 paxos A few questions: Could this be the cause of the slow addition/synchronization? Is what looks like an unbounded number of osd_snaps expected? If trimming/compacting them would help, how would one do that? Thanks, Brian ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

1 0

Errors handle_connect_reply_2 connect got BADAUTHORIZER
by Thomas 23 Sep '19

23 Sep '19

Hi, this is another issue that I'm facing with my current Ceph cluster (2x MDS, 6x ODS). When I try to delete a RBD image the CLI is flooded with error messages: root@ld3955:# rbd rm hdd/vm-304-disk-0 Removing image: 3% complete...2019-09-23 14:25:01.954 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb13403f1d0 0x7fb1340416e0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.954 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144076720 0x7fb144078c00 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.958 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb14407ca70 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.958 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144076720 0x7fb144078ba0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 6% complete...2019-09-23 14:25:01.958 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079460 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 7% complete...2019-09-23 14:25:01.958 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144076720 0x7fb144078ba0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.958 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079460 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.958 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144076720 0x7fb144078ba0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 8% complete...2019-09-23 14:25:01.958 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079460 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 9% complete...2019-09-23 14:25:01.962 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079dc0 0x7fb144078730 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 10% complete...2019-09-23 14:25:01.962 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079440 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.962 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079da0 0x7fb144078730 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 12% complete...2019-09-23 14:25:01.962 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079440 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.962 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079da0 0x7fb144078730 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 13% complete...2019-09-23 14:25:01.962 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079440 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 14% complete...2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c80 0x7fb144078730 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 15% complete...2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c60 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 16% complete...2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 17% complete...2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c60 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.966 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c60 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 19% complete...2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c60 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 20% complete...2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c60 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 21% complete...2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 22% complete...2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c60 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb1440793b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 24% complete...2019-09-23 14:25:01.970 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c40 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.970 7fb15bfcb700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb1340ba610 0x7fb1340bca70 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 25% complete...2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079350 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407ea30 0x7fb144081410 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15bfcb700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb144079c40 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079350 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 26% complete...2019-09-23 14:25:01.974 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb14407ea30 0x7fb144080fc0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079c40 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407ea30 0x7fb144082c20 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb14407a490 0x7fb144079350 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079310 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb144079bf0 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 27% complete...2019-09-23 14:25:01.974 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb14407a490 0x7fb144079310 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407ea30 0x7fb144082c20 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 28% complete...2019-09-23 14:25:01.974 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb144079bb0 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb14407a490 0x7fb144079310 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2019-09-23 14:25:01.974 7fb15b7ca700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.96:6833/12573 conn(0x7fb14407ea30 0x7fb144082c20 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER Removing image: 29% complete...2019-09-23 14:25:01.978 7fb15c7cc700 0 --1- 10.97.206.91:0/3920420758 >> v1:10.97.206.93:7029/14894 conn(0x7fb144079bb0 0x7fb144078700 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER How can I fix this error? THX

1 0

vfs_ceph and permissions
by ceph-users＠dxps31.33mail.com 23 Sep '19

23 Sep '19

Hi ceph-users, I'm having problems with ACLs and vfs_ceph. I'm not sure this is the right list for those questions, please direct me elsewhere or just tell me to bugger off if I'm in the wrong place. I am running a Samba 4.7 domain controller connecting to a nautilus ceph cluster. I am using proxmox for the ceph cluster and Zentyal for managing the samba domain. I am not able to set the permissions for the share within Computer Management My smb.conf share settings block is: [data2] browseable = yes force create mode = 0660 force directory mode = 0660 valid users = @"Domain Users", @"Domain Admins", @"Domain Admins" read list = write list = @"Domain Users", @"Domain Admins" admin users = @"Domain Admins" vfs objects = acl_xattr full_audit full_audit:failure = connect opendir disconnect unlink mkdir rmdir open rename map acl inherit = yes store dos attributes = yes vfs objects = ceph path = /new/ kernel share modes = no ceph:config_file = /etc/ceph/ceph.conf ceph:user_id = admin (I know - I will shange the user from admin as soon as I have a working setup!) The settings are similar share (set up by the Zentyal UI) that allows me to set permissions correctly: [test] path = /home/samba/shares/test browseable = yes force create mode = 0660 force directory mode = 0660 valid users = @"Domain Admins", @"Domain Admins" read list = write list = @"Domain Admins" admin users = @"Domain Admins" vfs objects = acl_xattr full_audit recycle full_audit:failure = connect opendir disconnect unlink mkdir rmdir open rename When trying to, for instance, add full control permissions for a user in Computer Management I get "Failed to enumerate objects in the container. Access is denied." / "Unable to change permission changes [..] Access denied.". I am logged in as administrator when trying to change the permissions and this is (an extract) of the error in the samba.log [2019/09/07 15:57:14.656361, 2] ../source3/smbd/posix_acls.c:3008(set_canon_ace_list) set_canon_ace_list: sys_acl_set_file type file failed for file . (Operation not supported). [2019/09/07 15:57:14.656435, 3] ../source3/smbd/posix_acls.c:3093(convert_canon_ace_to_posix_perms) convert_canon_ace_to_posix_perms: Too many ACE entries for file . to convert to posix perms. [2019/09/07 15:57:14.656472, 3] ../source3/smbd/posix_acls.c:3986(set_nt_acl) set_nt_acl: failed to convert file acl to posix permissions for file .. [2019/09/07 15:57:14.656511, 3] ../source3/smbd/smb2_server.c:3139(smbd_smb2_request_error_ex) smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] || at ../source3/smbd/smb2_setinfo.c:132 [2019/09/07 15:57:18.042643, 3] ../source3/smbd/nttrans.c:2036(smbd_do_query_security_desc) smbd_do_query_security_desc: sd_size = 120. [2019/09/07 15:57:18.043643, 3] ../source3/smbd/nttrans.c:2036(smbd_do_query_security_desc) smbd_do_query_security_desc: sd_size = 76. [2019/09/07 15:57:18.045086, 2] ../source3/smbd/posix_acls.c:3008(set_canon_ace_list) set_canon_ace_list: sys_acl_set_file type file failed for file . (Operation not supported). [2019/09/07 15:57:18.045171, 3] ../source3/smbd/posix_acls.c:3093(convert_canon_ace_to_posix_perms) convert_canon_ace_to_posix_perms: Too many ACE entries for file . to convert to posix perms. [2019/09/07 15:57:18.045208, 3] ../source3/smbd/posix_acls.c:3986(set_nt_acl) set_nt_acl: failed to convert file acl to posix permissions for file .. [2019/09/07 15:57:18.045248, 3] ../source3/smbd/smb2_server.c:3139(smbd_smb2_request_error_ex) smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] || at ../source3/smbd/smb2_setinfo.c:132 [2019/09/07 15:57:19.016012, 3] ../source3/winbindd/winbindd_misc.c:395(winbindd_interface_version) [ 6733]: request interface version (version = 29) [2019/09/07 15:57:19.016349, 3] ../source3/winbindd/winbindd_misc.c:428(winbindd_priv_pipe_dir) Any pointers would be great!

3 5

V/v [Ceph] problem with delete object in large bucket
by tuan dung 23 Sep '19

23 Sep '19

Hi Ceph team, Can you explain for me how ceph deleting object work? I have a bucket with above 100M object (file size ~ 50KB). When I delete object for free space, speed of deleting object very slow (about ~ 30-33 objects /s). I wan to tuning performance of cluster but i do not clearly know how ceph do deleting objects? Thank you very much ----------------------------- Br, Dương Tuấn Dũng

2 2

Cannot start virtual machines KVM / LXC
by Thomas 23 Sep '19

23 Sep '19

Hi, here I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS). The issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds. All these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd This pool hdd is relatively small compared to the largest pool: hdb_backup root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B hdb_backup 589 TiB 51262212 0 153786636 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 B 0 B hdd 3.2 TiB 281884 6568 845652 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 B 0 B pve_cephfs_data 955 GiB 91832 0 275496 0 0 3038 2103 1021 MiB 102170 318 GiB 0 B 0 B pve_cephfs_metadata 486 MiB 62 0 186 0 0 7 860 1.4 GiB 12393 166 MiB 0 B 0 B total_objects 51635990 total_used 597 TiB total_avail 522 TiB total_space 1.1 PiB This is the current health status of the ceph cluster: cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 87 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 54 pgs inactive, 47 pgs peering, 1 pg stale Degraded data redundancy: 129598/154907946 objects degraded (0.084%), 33 pgs degraded, 33 pgs undersized Degraded data redundancy (low space): 322 pgs backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 1 pools have too many placement groups 21 slow requests are blocked > 32 sec services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby osd: 360 osds: 356 up, 356 in; 382 remapped pgs data: pools: 5 pools, 8868 pgs objects: 51.64M objects, 197 TiB usage: 597 TiB used, 522 TiB / 1.1 PiB avail pgs: 0.056% pgs unknown 0.553% pgs not active 129598/154907946 objects degraded (0.084%) 2211119/154907946 objects misplaced (1.427%) 8458 active+clean 298 active+remapped+backfill_toofull 29 remapped+peering 24 active+undersized+degraded+remapped+backfill_toofull 22 active+remapped+backfill_wait 17 peering 5 unknown 5 active+recovery_wait+undersized+degraded+remapped 3 active+undersized+degraded+remapped+backfill_wait 2 activating+remapped 1 active+clean+remapped 1 stale+peering 1 active+remapped+backfilling 1 active+recovering+undersized+remapped 1 active+recovery_wait+degraded io: client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr I believe the cluster is busy with rebalancing pool hdb_backup. I set the balance mode upmap recently after the 589TB data was written. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } In order to resolve the issue with pool hdd I started some investigation. First step was to install drivers for the NIC provided Mellanox. Then I configured some kernel parameters recommended <https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox. However this didn't fix the issue. In my opinion I must get rid of all "slow requests are blocked". When I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd. This means none of the disks belonging to pool hdb_backup is showing a comparable behaviour. Then I checked the running processes on the different OSD nodes; I use tool "glances" here. Here I can see single processes that are running for hours and consuming much CPU, e.g. 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S 41M 2K /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S 14M 3K /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph --setgroup ceph Similar processes are running on 4 OSD nodes. All processes have in common that the relevant OSD belongs to pool hdd. Furthermore glances gives me this alert: CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, ceph-osd, ceph-osd What can / should I do now? Kill the long running processes? Stop the relevant OSDs? Please advise? THX Thomas

4 9

Ceph FS not releasing space after file deletion
by Gustavo Tonini 23 Sep '19

23 Sep '19

Dear ceph users, we are using CEPH since Jewel and now (with Mimic) are facing a problem that is impeding us to use it appropriately. After deleting some big backup files, we have noticed that their space has not returned to the pool's free space. Considering the mentioned pool, du shows that we have a total of 247TB allocated in files and "ceph df" says we have 293TB of used space. How can we debug this problem and force ceph to release unallocated space? Gustavo.

5 9

Binding to cluster-addr
by 徐蕴 22 Sep '19

22 Sep '19

Hello Ceph Users, I deployed a ceph cluster (v14.2.1, using docker), the cluster status seems OK but the write performance tested by rados bench seems bad. When I check the network connection using netstat -nat, I found that there is no connection established using the cluster network interface. The OSD daemon entrypoint is something like this: /usr/bin/ceph-osd -f --public-addr 111.111.121.2 --cluster-addr 111.111.131.2 -i 1 But there is no connection established using the IP 111.111.131.2. Is it a problematic behavior? If yes, would you please give me some hints to debug this problem? Thank you! Xu Yun

1 1

Identify rbd snapshot
by Marc Roos 21 Sep '19

21 Sep '19

I have this error. I have found the rbd image with the block_name_prefix:1f114174b0dc51, how can identify what snapshot this is? (Is it a snapshot?) 2019-08-29 16:16:49.255183 7f9b3f061700 -1 log_channel(cluster) log [ERR] : deep-scrub 17.36 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head : expected clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 missing 2019-08-29 16:24:54.806912 7f9b3f061700 -1 log_channel(cluster) log [ERR] : 17.36 deep-scrub 1 errors

2 1

RGW backup to tape
by Robert LeBlanc 20 Sep '19

20 Sep '19

The question was posed, "What if we want to backup our RGW data to tape?" Anyone doing this? Any suggestions? We could probably just catch any PUT requests and queue them to be written to tape. Our dataset is so large, that traditional backup solutions don't seem feasible (GFS), so probably a single copy (or two copies on different tapes at the same time) when the object is created. Bonus points for being near-line. Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

3 3