- ceph-users - lists.ceph.io

Re: RadosGW cant list objects when there are too many of them
by Paul Emmerich 21 Oct '19

21 Oct '19

Listing large buckets is slow due to S3 ordering requirements, it's approximately O(n^2). However, I wouldn't consider 5M to be a large bucket, it should go to only ~50 shards which should still perform reasonable. How fast are your metadata OSDs? Try --allow-unordered in radosgw-admin to get an unordered result which is only O(n) as you'd expect. For boto3: I'm not sure if v2 object listing is available yet (I think it has only been merged into master but has not yet made it into a release?). It doesn't support unordered listing but there has been some work to implement it there, not sure about the current state. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Thu, Oct 17, 2019 at 9:19 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > > Dear All > > I have a bucket with 5 million Objects and I cant list objects with > radosgw-admin bucket list --bucket=bucket | jq .[].name > or listing files using boto3 > > s3 = boto3.client('s3', > endpoint_url=credentials['endpoint_url'], > aws_access_key_id=credentials['access_key'], > aws_secret_access_key=credentials['secret_key']) > > response = s3.list_objects_v2(Bucket=bucket_name) > for item in response['Contents']: > print(item['Key']) > > what is the solution ? how can I find list of my objects ? > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Getting after upgrade to nautilus every few seconds: cluster [DBG] pgmap
by Marc Roos 20 Oct '19

20 Oct '19

Is it really necessary to get these every few seconds? mgr.b (mgr.3714636) 264 : cluster [DBG] pgmap v265: 384

1 0

Re: Module 'rbd_support' has failed: Not found or unloadable
by Jason Dillaman 20 Oct '19

20 Oct '19

What are the caps for your mgr user? Does it have permissions to talk to the OSDs? # ceph auth get mgr.<XYZ> On Sat, Oct 19, 2019 at 6:58 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > > 2019-10-19 22:26:12.606 7fe16448cb80 1 mgr[py] Loading python module > 'rbd_support' > 2019-10-19 22:26:14.627 7fe13b494700 -1 mgr load Failed to construct > class in 'rbd_support' > File "/usr/share/ceph/mgr/rbd_support/module.py", line 1326, in > __init__ > File "/usr/share/ceph/mgr/rbd_support/module.py", line 610, in > __init__ > File "/usr/share/ceph/mgr/rbd_support/module.py", line 674, in > init_task_queue > File "/usr/share/ceph/mgr/rbd_support/module.py", line 708, in > load_task_queue > 2019-10-19 22:26:14.628 7fe13b494700 -1 mgr operator() Failed to run > module in active mode ('rbd_support') > > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] Module 'rbd_support' has failed: Not found or > unloadable > > > After luminous -> nautilus upgrade I have this error, how to resolve? > > Tried enabling and disabling it. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Jason

3 3

Module 'rbd_support' has failed: Not found or unloadable
by Marc Roos 19 Oct '19

19 Oct '19

After luminous -> nautilus upgrade I have this error, how to resolve? Tried enabling and disabling it.

1 1

Luminous -> nautilus upgrade on centos7 lots of Unknown lvalue logs
by Marc Roos 19 Oct '19

19 Oct '19

Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' I am getting lots these in the log files after updating the rpms Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service'

1 0

ceph balancer do not start
by Jan Peters 18 Oct '19

18 Oct '19

Hello, i use ceph 12.2.12 and would like to activate the ceph balancer. unfortunately no redistribution of the PGs is started: ceph balancer status { "active": true, "plans": [], "mode": "crush-compat" } ceph balancer eval current cluster score 0.023776 (lower is better) ceph config-key dump { "initial_mon_keyring": "AQBLchlbAAAAABAA+5CuVU+8MB69xfc3xAXkjQ==", "mgr/balancer/active": "1", "mgr/balancer/max_misplaced:": "0.01", "mgr/balancer/mode": "crush-compat" } What am I not doing correctly? best regards

1 0

CephFS and 32-bit Inode Numbers
by Dan van der Ster 18 Oct '19

18 Oct '19

Hi all, One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3. I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3. Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?) Thanks! Dan

7 8

OSD node suddenly slow to responding cmd
by Amudhan P 18 Oct '19

18 Oct '19

Hi, I am using Ceph nautilus cluster and i found one of my OSD node running 3 OSD's service suddenly went down and it was very slow typing command. I killed ceph-osd process and system become normal and started all OSD service. After that it becomes normal I figure out that due low memory the system was slow but there is no OOM in any of the logs. I can see heartbeat_check and suicide time out in below msg in OSD logs, but why did this happened? what could be the issue? *osd.1* 2019-10-16 20:49:38.433 7f246f302700 0 bluestore(/var/lib/ceph/osd/ceph-1) log_latency_fn slow operation observed for _txc_committed_kv, latency = 6.20965s, txc = 0x55 a2541f1500 2019-10-16 20:50:05.233 7f247bf7d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f245e4e6700' had timed out after 15 2019-10-16 20:50:06.721 7f247af7b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f245e4e6700' had timed out after 15 2019-10-16 20:53:57.881 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.113:6805 osd.2 since back 2019-10-16 20:49:44.711088 front 2019-10-16 20:4 9:47.684146 (oldest deadline 2019-10-16 20:50:07.937387) 2019-10-16 20:55:26.997 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.111:6821 osd.3 since back 2019-10-16 20:49:44.711031 front 2019-10-16 20:4 9:44.711495 (oldest deadline 2019-10-16 20:50:07.937387) 2019-10-16 20:56:25.513 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.112:6805 osd.4 since back 2019-10-16 20:49:44.484031 front 2019-10-16 20:4 9:44.711337 (oldest deadline 2019-10-16 20:50:03.162338) 2019-10-16 20:57:52.557 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.113:6813 osd.5 since back 2019-10-16 20:49:44.484064 front 2019-10-16 20:4 9:44.711504 (oldest deadline 2019-10-16 20:50:03.162338) 2019-10-16 20:59:22.837 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.111:6805 osd.6 since back 2019-10-16 20:49:44.711483 front 2019-10-16 20:4 9:44.483337 (oldest deadline 2019-10-16 20:50:03.162338) 2019-10-16 21:01:18.761 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.112:6821 osd.7 since back 2019-10-16 20:49:44.711275 front 2019-10-16 20:4 9:44.711352 (oldest deadline 2019-10-16 20:50:07.937387) 2019-10-16 21:02:34.021 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.113:6821 osd.8 since back 2019-10-16 20:49:44.711517 front 2019-10-16 20:4 9:44.711470 (oldest deadline 2019-10-16 20:50:07.937387) 2019-10-16 21:04:26.361 7f247b77c700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f245d4e4700' had timed out after 60 2019-10-16 21:04:46.225 7f247b77c700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f245e4e6700' had timed out after 15 2019-10-16 21:05:01.453 7f247b77c700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f245e4e6700' had suicide timed out after 150 2019-10-16 21:34:06.365 7f0a6105cf80 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-10-16 21:34:06.365 7f0a6105cf80 0 ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable), process ceph-osd, pid 203416 2019-10-16 21:34:06.365 7f0a6105cf80 0 pidfile_write: ignore empty --pid-file *osd.4* 2019-10-16 20:53:57.881 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.112:6813 osd.1 since back 2019-10-16 20:49:41.268550 front 2019-10-16 20:49:41.268546 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 20:55:26.997 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.113:6805 osd.2 since back 2019-10-16 20:49:44.484184 front 2019-10-16 20:49:44.483961 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 20:56:25.513 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.111:6821 osd.3 since back 2019-10-16 20:49:41.192151 front 2019-10-16 20:49:41.191856 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 20:57:52.557 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.113:6813 osd.5 since back 2019-10-16 20:49:44.484278 front 2019-10-16 20:49:47.920080 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 20:59:22.837 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.111:6805 osd.6 since back 2019-10-16 20:49:41.268846 front 2019-10-16 20:49:41.192139 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 21:01:16.561 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.112:6821 osd.7 since back 2019-10-16 20:49:41.380152 front 2019-10-16 20:49:41.380178 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 21:02:15.621 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.113:6821 osd.8 since back 2019-10-16 20:49:43.862767 front 2019-10-16 20:49:44.484153 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 21:04:14.997 7f7efba77700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7edd7df700' had timed out after 60 2019-10-16 21:04:31.689 7f7efba77700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7eddfe0700' had timed out after 60 2019-10-16 21:04:56.025 7f7efba77700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7ede7e1700' had timed out after 60 2019-10-16 21:05:10.409 7f7efc278700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7edd7df700' had timed out after 60 2019-10-16 21:05:20.813 7f7efc278700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7eddfe0700' had timed out after 60 2019-10-16 21:05:34.149 7f7efc278700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7ede7e1700' had timed out after 60 2019-10-16 21:05:48.465 7f7efb276700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7edd7df700' had timed out after 60 2019-10-16 21:06:15.409 7f7efb276700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7eddfe0700' had timed out after 60 2019-10-16 21:06:32.173 7f7efb276700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7ede7e1700' had timed out after 60 2019-10-16 21:21:53.125 7f7ee87f5700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2019-10-16 21:22:15.137 7f7ee87f5700 4 rocksdb: [db/db_impl.cc:778] *osd.7* 2019-10-16 20:53:57.881 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.112:6813 osd.1 since back 2019-10-16 20:49:41.380241 front 2019-10-16 20:49:41.380043 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 20:55:26.997 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.113:6805 osd.2 since back 2019-10-16 20:49:44.484063 front 2019-10-16 20:49:47.920022 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 20:56:25.513 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.111:6821 osd.3 since back 2019-10-16 20:49:41.379978 front 2019-10-16 20:49:41.380217 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 20:57:49.685 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.112:6805 osd.4 since back 2019-10-16 20:49:41.380202 front 2019-10-16 20:49:41.380119 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 20:59:18.389 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.113:6813 osd.5 since back 2019-10-16 20:49:43.862826 front 2019-10-16 20:49:43.723800 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 21:01:16.561 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.111:6805 osd.6 since back 2019-10-16 20:49:41.380229 front 2019-10-16 20:49:41.380023 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 21:02:19.197 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.113:6821 osd.8 since back 2019-10-16 20:49:44.484017 front 2019-10-16 20:49:44.484078 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 21:21:53.125 7f959cbe1700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2019-10-16 21:22:20.221 7f959cbe1700 4 rocksdb: [db/db_impl.cc:778] *Journalctl - output* Oct 16 20:53:38 test-node2 systemd-resolved[1005]: Using degraded feature set (TCP) for DNS server 192.168.4.12. Oct 16 20:56:02 test-node2 ceph-osd[1710]: 2019-10-16 20:53:57.881 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.113:6805 osd.2 since back 2019- Oct 16 20:56:29 test-node2 ceph-osd[91197]: 2019-10-16 20:53:57.881 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.112:6813 osd.1 since back 2019 Oct 16 20:53:40 test-node2 systemd-resolved[1005]: Using degraded feature set (TCP) for DNS server 192.168.4.7. Oct 16 20:57:17 test-node2 ceph-osd[1713]: 2019-10-16 20:53:57.881 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.112:6813 osd.1 since back 2019- Oct 16 20:57:17 test-node2 ceph-osd[1713]: 2019-10-16 20:55:26.997 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.113:6805 osd.2 since back 2019- Oct 16 20:57:59 test-node2 ceph-osd[1710]: 2019-10-16 20:55:26.997 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.111:6821 osd.3 since back 2019- Oct 16 20:57:59 test-node2 ceph-osd[1710]: 2019-10-16 20:56:25.513 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.112:6805 osd.4 since back 2019- Oct 16 20:58:43 test-node2 ceph-osd[91197]: 2019-10-16 20:55:26.997 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.113:6805 osd.2 since back 2019 Oct 16 20:58:43 test-node2 ceph-osd[91197]: 2019-10-16 20:56:25.513 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.111:6821 osd.3 since back 2019 Oct 16 20:53:51 test-node2 systemd-networkd[907]: enp5s0: Failed to save LLDP data to /run/systemd/netif/lldp/2: No space left on device Oct 16 21:00:13 test-node2 ceph-osd[1713]: 2019-10-16 20:56:25.513 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.111:6821 osd.3 since back 2019- Oct 16 21:00:13 test-node2 ceph-osd[1713]: 2019-10-16 20:57:52.557 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.113:6813 osd.5 since back 2019- Oct 16 21:01:42 test-node2 ceph-osd[1710]: 2019-10-16 20:57:52.557 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.113:6813 osd.5 since back 2019- Oct 16 21:01:42 test-node2 ceph-osd[1710]: 2019-10-16 20:59:22.837 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.111:6805 osd.6 since back 2019- Oct 16 21:02:33 test-node2 ceph-osd[91197]: 2019-10-16 20:57:49.685 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.112:6805 osd.4 since back 2019 Oct 16 21:02:33 test-node2 ceph-osd[91197]: 2019-10-16 20:59:18.389 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.113:6813 osd.5 since back 2019 Oct 16 20:54:24 test-node2 systemd-networkd[907]: enp5s0: Failed to save LLDP data to /run/systemd/netif/lldp/2: No space left on device Oct 16 21:04:08 test-node2 ceph-osd[1713]: 2019-10-16 20:59:22.837 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.111:6805 osd.6 since back 2019- Oct 16 21:04:08 test-node2 ceph-osd[1713]: 2019-10-16 21:01:16.561 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.112:6821 osd.7 since back 2019- Oct 16 21:04:54 test-node2 ceph-osd[1710]: 2019-10-16 21:01:18.761 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.112:6821 osd.7 since back 2019- Oct 16 21:04:54 test-node2 ceph-osd[1710]: 2019-10-16 21:02:34.021 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.113:6821 osd.8 since back 2019- Oct 16 21:05:55 test-node2 ceph-osd[91197]: 2019-10-16 21:01:16.561 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.111:6805 osd.6 since back 2019 Oct 16 21:05:55 test-node2 ceph-osd[91197]: 2019-10-16 21:02:19.197 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.113:6821 osd.8 since back 2019 Oct 16 20:54:56 test-node2 systemd-networkd[907]: enp5s0: Failed to save LLDP data to /run/systemd/netif/lldp/2: No space left on device Oct 16 21:07:37 test-node2 ceph-osd[1713]: 2019-10-16 21:02:15.621 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.113:6821 osd.8 since back 2019- Oct 16 21:08:12 test-node2 ceph-osd[1710]: *** Caught signal (Aborted) ** Oct 16 21:08:12 test-node2 ceph-osd[1710]: in thread 7f245e4e6700 thread_name:tp_osd_tp Oct 16 21:11:49 test-node2 ceph-osd[1710]: /build/ceph-14.2.4/src/common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(const ceph::heartbeat_handle_d*, Oct 16 21:11:49 test-node2 ceph-osd[1710]: /build/ceph-14.2.4/src/common/HeartbeatMap.cc: 82: ceph_abort_msg("hit suicide timeout") Oct 16 21:12:36 test-node2 ceph-osd[1710]: ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) Oct 16 21:12:36 test-node2 ceph-osd[1710]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato Oct 16 21:34:05 test-node2 systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT Oct 16 21:34:05 test-node2 systemd[1]: systemd-journald.service: Failed with result 'watchdog'. Oct 16 21:34:05 test-node2 systemd[1]: ceph-osd(a)1.service: Main process exited, code=killed, status=6/ABRT Oct 16 21:34:05 test-node2 systemd[1]: ceph-osd(a)1.service: Failed with result 'signal'. Oct 16 21:34:05 test-node2 systemd[1]: systemd-journald.service: Service has no hold-off time, scheduling restart. Oct 16 21:34:05 test-node2 systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. Oct 16 21:34:05 test-node2 systemd[1]: ceph-osd(a)4.service: Main process exited, code=killed, status=9/KILL Oct 16 21:34:05 test-node2 systemd[1]: ceph-osd(a)4.service: Failed with result 'signal'. Oct 16 21:34:05 test-node2 systemd[1]: ceph-osd(a)1.service: Service hold-off time over, scheduling restart. Oct 16 21:34:05 test-node2 systemd[1]: ceph-osd(a)1.service: Scheduled restart job, restart counter is at 1. Oct 16 21:34:05 test-node2 systemd[1]: Started Ceph object storage daemon osd.4. Oct 16 21:34:05 test-node2 systemd[1]: Started Ceph object storage daemon osd.1. Oct 16 21:34:05 test-node2 systemd[1]: Started Ceph object storage daemon osd.7. Oct 17 06:25:03 test-node2 ceph-osd[203415]: 2019-10-17 06:25:03.652 7f91880ca700 -1 Fail to open '/proc/204405/cmdline' error = (2) No such file or directory Oct 17 06:25:03 test-node2 ceph-osd[203415]: 2019-10-17 06:25:03.680 7f91880ca700 -1 received signal: Hangup from <unknown> (PID: 204405) UID: 0 Oct 17 06:25:03 test-node2 ceph-osd[203415]: 2019-10-17 06:25:03.736 7f91880ca700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph- Oct 17 06:25:04 test-node2 ceph-osd[203417]: 2019-10-17 06:25:03.652 7f2ce27ac700 -1 Fail to open '/proc/204405/cmdline' error = (2) No such file or directory Oct 17 06:25:04 test-node2 ceph-osd[203417]: 2019-10-17 06:25:03.692 7f2ce27ac700 -1 received signal: Hangup from <unknown> (PID: 204405) UID: 0 Oct 17 06:25:04 test-node2 ceph-osd[203417]: 2019-10-17 06:25:03.736 7f2ce27ac700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph- Oct 17 06:25:04 test-node2 ceph-osd[203416]: 2019-10-17 06:25:03.960 7f0a577ab700 -1 Fail to open '/proc/204406/cmdline' error = (2) No such file or directory Oct 17 06:25:04 test-node2 ceph-osd[203416]: 2019-10-17 06:25:03.960 7f0a577ab700 -1 received signal: Hangup from <unknown> (PID: 204406) UID: 0 Oct 17 06:25:04 test-node2 ceph-osd[203416]: 2019-10-17 06:25:03.960 7f0a577ab700 -1 Fail to open '/proc/204406/cmdline' error = (2) No such file or directory Oct 17 06:25:04 test-node2 ceph-osd[203416]: 2019-10-17 06:25:03.960 7f0a577ab700 -1 received signal: Hangup from <unknown> (PID: 204406) UID: 0

1 0

Re: Change device class in EC profile
by Frank Schilder 18 Oct '19

18 Oct '19

Hi Maks, thanks for looking at this. Unfortunately, this does not answer the question. After steps 1-3 you are exactly in the same situation as I am, namely that the profile attached to the pool is outdated and, therefore, contains invalid information that will be confusing. To check, execute ceph osd pool get POOL_NAME erasure_code_profile and then do ceph osd erasure-code-profile get PROFILE_NAME_RETURNED_ABOVE This will not match the new crush rule unless there was no change. Note that you cannot *set* the EC profile on a pool to match the profile you created your new crush rule from. Hence, the pool info will drag outdated information along. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Maks Kowalik <maks_kowalik(a)poczta.fm> Sent: 18 October 2019 14:01 To: Frank Schilder Subject: Re: [ceph-users] Change device class in EC profile I managed to achieve a similar goal (old EC profile didn't use classes at all) by: 1. Creating a new EC profile. 2. Creating new crush rule based on the EC profile. 3. Changing crush rule for a pool to a newly created one. Kind regards, Maks pt., 18 paź 2019 o 13:07 Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> napisał(a): I recently moved an EC pool from HDD to SSD by changing the device class in the crush rule. I would like to complete this operation by cleaning up a dirty trail. The EC profile attached to this pool is called sr-ec-6-2-hdd and it is easy enough to rename that to sr-ec-6-2-ssd. However, the profile itself contains the device class as well: crush-device-class=hdd [...] I can already see the confusion this will cause in the future. Is this situation one of the few instances where using the --force option is warranted to change the device class of a profile, as in: osd erasure-code-profile set sr-ec-6-2-ssd crush-device-class=ssd --force If not, how can I change the device class of the profile? Many thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

iscsi gate install
by Fyodor Ustinov 18 Oct '19

18 Oct '19

Hi! CEPH documentation requre "tcmu-runner-1.4.0 or newer package", but I can not find this package for Centos. Maybe someone knows where to download this package? WBR, Fyodor.

2 2