- ceph-users - lists.ceph.io

Advice on sizing WAL/DB cluster for Optane and SATA SSD disks.
by Victor Hooi 16 Mar '20

16 Mar '20

Hi, I'm building a 4-node Proxmox cluster, with Ceph for the VM disk storage. On each node, I have: - 1 x 512Gb M.2 SSD (for Proxmox/boot volume) - 1 x 960GB Intel Optane 905P (for Ceph WAL/DB) - 6 x 1.92TB Intel S4610 SATA SSD (for Ceph OSD) I'm using the Proxmox "pveceph" command to setup the OSDs. By default this seems to pick 10% of the OSD size for the DB volume, and 1% of the OSD size for the WAL volume. This means after four drives, I ran out of space: # pveceph osd create /dev/sde -db_dev /dev/nvme0n1 > create OSD on /dev/sde (bluestore) > creating block.db on '/dev/nvme0n1' > Rounding up size to full physical extent 178.85 GiB > lvcreate > 'ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee/osd-db-da591d0f-8a05-42fa-bc62-a093bf98aded' > error: Volume group "ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee" has > insufficient free space (45784 extents): 45786 required. Anyway, I assume that means I need to tune my DB and WAL volumes down from the defaults. What advice to you have in terms of making best use of the available space, between WAL and DB? What is the impact of having WAL and DB smaller than 1% and 10% of OSD size respectively? Thanks, Victor

5 5

HEALTH_WARN 1 pools have too few placement groups
by Dietmar Rieder 16 Mar '20

16 Mar '20

Hi, I was planing to activate the pg_autoscaler on a EC (6+3) pool which I created two years ago. Back then I calculated the total # of pgs for this pool with a target per ods pg # of 150 (this was the recommended /osd pg number as far as I recall). I used the RedHat ceph pg per pool calculator [1] with the following settings: PoolyType: EC K: 6 M: 3 OSD #: 220 %Data: 100.00 Target PGs per OSD: 150 This resulted into a suggested PG count of 4096, which I used to create that pool. (BTW: I get the same using the ceph.io pgcalc [2], setting size to 9 (=EC 6+3)) # ceph osd df shows that I have now 162-169 pgs / osd Now I enabled the pg_autoscaler and set autoscale_mode to "warn", using the following commands: # ceph mgr module enable pg_autoscaler # ceph osd pool set hdd-ec-data-pool pg_autoscale_mode warn when i set the target_ratio_size to 1, using: # ceph osd pool set hdd-ec-data-pool target_size_ratio 1 I get the warning: 1 pools have too few placement groups The autoscale status shows that pg number in the EC pool would get scaled up to 16384 pgs: # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE ssd-rep-metadata-pool 11146M 3.0 29805G 0.0011 1.0 64 4 off ssd-rep-data-pool 837.8G 3.0 29805G 0.0843 1.0 1024 64 off hdd-ec-data-pool 391.3T 1.5 1614T 0.3636 1.0000 1.0 4096 16384 warn How does this relate the the calculated number of 4096 pgs for that EC pool? I mean 16384 is quite a jump from the original value. Is the autoscaler miscalculating something or is the pg calculator wrong? Can someone explain it? (My cluster is on ceph version 14.2.7) Thanks so much Dietmar [1]: https://access.redhat.com/labs/cephpgc/ [2]: https://ceph.io/pgcalc/ -- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Email: dietmar.rieder(a)i-med.ac.at Web: http://www.icbi.at

2 2

Zabbix module failed to send data - SSL support
by tdados＠hotmail.com 16 Mar '20

16 Mar '20

Hello all, We are having an issue with the Ceph Zabbix module and it's failing to send data. The reason is that in our Zabbix infrastructure we use Encryption and agent connection with certificate as well. I see the logs that are failing due to that reason from the zabbix proxy servers. 1329:20200312:092834.705 connection of type "unencrypted" is not allowed for host "xxx" - this is ceph host 1328:20200312:092840.337 connection of type "unencrypted" is not allowed for host "xxx" 1329:20200312:092909.423 connection of type "unencrypted" is not allowed for host "xxx" In the zabbix ceph config from ceph there is no configuration option to allow ssl. I also tried different options with the sender like to use the configuration file like ""ceph zabbix config-set zabbix_sender "/usr/bin/zabbix_sender -c /etc/zabbix/zabbix_agentd.conf"" but didn't work either. Anyone has any tips how to tell from ceph config to allow/enable ssl on the zabbix module?

1 0

Trying to follow installation documentation
by Mark M 16 Mar '20

16 Mar '20

Hi List, Trying to learn Ceph, so I'm using a Vagrant setup of six nodes with Docker and LVM on them. But as soon as I reach this step https://docs.ceph.com/docs/master/cephadm/#bootstrap-a-new-cluster I get stuck with this error: ``` INFO:cephadm:Mgr epoch 5 not available, waiting (6/10)... INFO:cephadm:Non-zero exit code 22 from /bin/docker run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-master-devel -e NODE_NAME=monitor01 -v /var/log/ceph/114dc730-66ca -11ea-b6f9-5254008afee6:/var/log/ceph:z -v /tmp/ceph-tmpbrsg0raa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq7q4mc5h:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph /daemon-base:latest-master-devel tell mgr mgr_status INFO:cephadm:/usr/bin/ceph:stderr no valid command found; 10 closest matches: INFO:cephadm:/usr/bin/ceph:stderr 0 INFO:cephadm:/usr/bin/ceph:stderr 1 INFO:cephadm:/usr/bin/ceph:stderr 2 INFO:cephadm:/usr/bin/ceph:stderr abort INFO:cephadm:/usr/bin/ceph:stderr assert INFO:cephadm:/usr/bin/ceph:stderr config diff INFO:cephadm:/usr/bin/ceph:stderr config diff get <var> INFO:cephadm:/usr/bin/ceph:stderr config get <var> INFO:cephadm:/usr/bin/ceph:stderr config help [<var>] INFO:cephadm:/usr/bin/ceph:stderr config set <var> <val>... INFO:cephadm:/usr/bin/ceph:stderr Error EINVAL: invalid command INFO:cephadm:Mgr epoch 5 not available, waiting (7/10)... ``` And then it just repeats. Also the "/var/log/ceph/114dc730-66ca-11ea-b6f9-5254008afee6" folder is empty, no logs. I must admit I'm stumped, I don't really know where to go from here. Any pointers for me perhaps? Thanks! Mark

1 1

Re: Kworker 100% with ceph-msgr (after upgrade to 14.2.6?)
by Ilya Dryomov 15 Mar '20

15 Mar '20

On Tue, Jan 14, 2020 at 10:31 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I think this is new since I upgraded to 14.2.6. kworker/7:3 100% > > [@~]# echo l > /proc/sysrq-trigger > > [Tue Jan 14 10:05:08 2020] CPU: 7 PID: 2909400 Comm: kworker/7:0 Not > tainted 3.10.0-1062.4.3.el7.x86_64 #1 > > [Tue Jan 14 10:05:08 2020] Workqueue: ceph-msgr ceph_con_workfn > [libceph] > [Tue Jan 14 10:05:08 2020] task: ffffa0d2cb9db150 ti: ffffa0d3040f0000 > task.ti: ffffa0d3040f0000 > [Tue Jan 14 10:05:08 2020] RIP: 0010:[<ffffffffb0192e7e>] > [<ffffffffb0192e7e>] generic_swap+0x1e/0x30 > [Tue Jan 14 10:05:08 2020] RSP: 0018:ffffa0d3040f3a20 EFLAGS: 00000206 > [Tue Jan 14 10:05:08 2020] RAX: 0000000000000072 RBX: 0000000000000060 > RCX: 0000000000000072 > [Tue Jan 14 10:05:08 2020] RDX: 0000000000000006 RSI: ffffa0c788914a7a > RDI: ffffa0c788914a42 > [Tue Jan 14 10:05:08 2020] RBP: ffffa0d3040f3a20 R08: 0000000000000028 > R09: 0000000000000016 > [Tue Jan 14 10:05:08 2020] R10: 0000000000000036 R11: ffffe94984224500 > R12: ffffa0c788914a40 > [Tue Jan 14 10:05:08 2020] R13: ffffffffc08d7da0 R14: ffffa0c788914a18 > R15: ffffa0c788914a78 > [Tue Jan 14 10:05:08 2020] FS: 0000000000000000(0000) > GS:ffffa0d2cfbc0000(0000) knlGS:0000000000000000 > [Tue Jan 14 10:05:08 2020] CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > [Tue Jan 14 10:05:08 2020] CR2: 000055f0404d5e40 CR3: 0000001813010000 > CR4: 00000000000627e0 > [Tue Jan 14 10:05:08 2020] Call Trace: > [Tue Jan 14 10:05:08 2020] [<ffffffffb0193055>] sort+0x1c5/0x260 > [Tue Jan 14 10:05:08 2020] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d807b>] > build_snap_context+0x12b/0x290 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d820c>] > rebuild_snap_realms+0x2c/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d822b>] > rebuild_snap_realms+0x4b/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d91fc>] > ceph_update_snap_trace+0x3ec/0x530 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e2239>] > handle_reply+0x359/0xc60 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 > [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e56a>] ? > kernel_recvmsg+0x3a/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fcff4>] try_read+0x544/0x1300 > [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffafee13ce>] ? > account_entity_dequeue+0xae/0xd0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafee4d5c>] ? > dequeue_entity+0x11c/0x5e0 > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e417>] ? > kernel_sendmsg+0x37/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fdfb4>] > ceph_con_workfn+0xe4/0x1530 [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb057f568>] ? > __schedule+0x448/0x9c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebe21f>] > process_one_work+0x17f/0x440 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf336>] > worker_thread+0x126/0x3c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf210>] ? > manage_workers.isra.26+0x2a0/0x2a0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec61f1>] kthread+0xd1/0xe0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > [Tue Jan 14 10:05:08 2020] [<ffffffffb058cd37>] > ret_from_fork_nospec_begin+0x21/0x21 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > > > > [Tue Jan 14 10:05:08 2020] CPU: 7 PID: 2909400 Comm: kworker/7:0 Not > tainted 3.10.0-1062.4.3.el7.x86_64 #1 > > [Tue Jan 14 10:05:08 2020] Workqueue: ceph-msgr ceph_con_workfn > [libceph] > [Tue Jan 14 10:05:08 2020] task: ffffa0d2cb9db150 ti: ffffa0d3040f0000 > task.ti: ffffa0d3040f0000 > [Tue Jan 14 10:05:08 2020] RIP: 0010:[<ffffffffb0192200>] > [<ffffffffb0192200>] __x86_indirect_thunk_rax+0x0/0x20 > [Tue Jan 14 10:05:08 2020] RSP: 0018:ffffa0d3040f3a28 EFLAGS: 00000286 > [Tue Jan 14 10:05:08 2020] RAX: ffffffffb0192e60 RBX: 0000000000000010 > RCX: 0000000000000010 > [Tue Jan 14 10:05:08 2020] RDX: 0000000000000008 RSI: ffffa0d0e108b828 > RDI: ffffa0d0e108b818 > [Tue Jan 14 10:05:08 2020] RBP: ffffa0d3040f3a98 R08: 0000000000000000 > R09: 0000000000000016 > [Tue Jan 14 10:05:08 2020] R10: 0000000000000036 R11: ffffe949a9842280 > R12: ffffa0d0e108b818 > [Tue Jan 14 10:05:08 2020] R13: ffffffffc08d7da0 R14: ffffa0d0e108b818 > R15: ffffa0d0e108b828 > [Tue Jan 14 10:05:08 2020] FS: 0000000000000000(0000) > GS:ffffa0d2cfbc0000(0000) knlGS:0000000000000000 > [Tue Jan 14 10:05:08 2020] CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > [Tue Jan 14 10:05:08 2020] CR2: 000055f0404d5e40 CR3: 0000001813010000 > CR4: 00000000000627e0 > [Tue Jan 14 10:05:08 2020] Call Trace: > [Tue Jan 14 10:05:08 2020] [<ffffffffb0193055>] ? sort+0x1c5/0x260 > [Tue Jan 14 10:05:08 2020] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d807b>] > build_snap_context+0x12b/0x290 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d820c>] > rebuild_snap_realms+0x2c/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d822b>] > rebuild_snap_realms+0x4b/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d91fc>] > ceph_update_snap_trace+0x3ec/0x530 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e2239>] > handle_reply+0x359/0xc60 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 > [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e56a>] ? > kernel_recvmsg+0x3a/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fcff4>] try_read+0x544/0x1300 > [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffafee13ce>] ? > account_entity_dequeue+0xae/0xd0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafee4d5c>] ? > dequeue_entity+0x11c/0x5e0 > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e417>] ? > kernel_sendmsg+0x37/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fdfb4>] > ceph_con_workfn+0xe4/0x1530 [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb057f568>] ? > __schedule+0x448/0x9c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebe21f>] > process_one_work+0x17f/0x440 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf336>] > worker_thread+0x126/0x3c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf210>] ? > manage_workers.isra.26+0x2a0/0x2a0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec61f1>] kthread+0xd1/0xe0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > [Tue Jan 14 10:05:08 2020] [<ffffffffb058cd37>] > ret_from_fork_nospec_begin+0x21/0x21 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 Hi Marc, It's busy processing snapshot contexts, I think it's more or less the same issue that you reported in "ceph node crashed with these errors "kernel: ceph: build_snap_context" (maybe now it is urgent?)". Did "ceph: build_snap_context 100020c9287 ffff911a9a26bd00 fail -12" errors go away? How many snapshots do you have now? Did you file a tracker ticket? It's better to have all details in one place rather than scattered across different threads on the list. Adding Jeff and Zheng, who can take a closer look. Thanks, Ilya

2 2

Weird monitor and mgr behavior after update.
by Cassiano Pilipavicius 15 Mar '20

15 Mar '20

Hello, I have updated my cluster from luminous to nautilus and now my cluster is working but I am seeing a weird behavior in my monitors and managers. The monitors are using a huge amount of memory and becoming very slow. The CPU usage is also very higher than it used to be. The manager keeps constantly restarting and in many opportunities it report all zero values in the pg numbers, space usage etc. I also started receiving many slow ops warning from monitors. Does any one has seen something like this? Which info do I need to send about my cluster so I can get some help from more experienced users? This is one fo the weird log messages: 2020-03-15 06:26:39.474 7f0396cae700 -1 mon.cwvh2@0(leader) e27 get_health_metrics reporting 603796 slow ops, oldest is pool_op(create unmanaged snap pool 5 tid 1236847 name v382060) cluster: id: 4ed305d4-5847-4e63-b9bb-168361cf2e81 health: HEALTH_WARN 1/3 mons down, quorum cwvh8,cwvh15 services: mon: 3 daemons, quorum (age 4M), out of quorum: cwvh2, cwvh8, cwvh15 mgr: cwvh13(active, since 33s), standbys: cwvh14 osd: 100 osds: 100 up (since 17h), 100 in data: pools: 6 pools, 4160 pgs objects: 19.35M objects, 64 TiB usage: 132 TiB used, 94 TiB / 226 TiB avail pgs: 4154 active+clean 6 active+clean+scrubbing+deep io: client: 192 MiB/s rd, 52 MiB/s wr, 1.25k op/s rd, 1.34k op/s wr [root@cwvh15 ~]# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED backup 120 TiB 62 TiB 58 TiB 58 TiB 48.64 hdd 12 TiB 4.3 TiB 7.8 TiB 7.8 TiB 64.47 ssd 94 TiB 28 TiB 66 TiB 66 TiB 70.40 TOTAL 226 TiB 94 TiB 132 TiB 132 TiB 58.51 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL rbd 0 19 B 2 128 KiB 0 11 TiB CWVHDS 1 420 GiB 110.03k 848 GiB 39.36 653 GiB COMP 4 2.7 TiB 717.72k 5.5 TiB 81.19 653 GiB SSD 5 35 TiB 9.43M 66 TiB 88.08 4.4 TiB HDD 6 730 GiB 196.24k 1.4 TiB 52.84 653 GiB BKPR1 8 33 TiB 8.90M 58 TiB 56.73 22 TiB

1 0

How to get num ops blocked per OSD
by Robert LeBlanc 14 Mar '20

14 Mar '20

For Jewel I wrote a script to take the output of `ceph health detail --format=json` and send alerts to our system that ordered the osds based on how long the ops were blocked and which OSDs had the most ops blocked. This was really helpful to quickly identify which OSD out of a list of 100 would be the most probable one having issues. Since upgrading to Luminous, I don't get that and I'm not sure where that info went to. Do I need to query the manager now? This is the regex I was using to extract the pertinent information: '^(\d+) ops are blocked > (\d+\.+\d+) sec on osd\.(\d+)$' Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 2

rgw.none shows extremely large object count
by yuezhu3＠gmail.com 14 Mar '20

14 Mar '20

Hello, we are running a ceph cluster + rgw on luminous 12.2.12 that serves as a S3 compatible storage. We have noticed some buckets where the `rgw.none` from the output of `radosgw-admin bucket stats` shows extremely large value for `num_objects`, which is not convincible. It does look like an underflow by subtracting a positive number from 0 and then the value is interpreted and shown as an uint64. For example, ``` # radosgw-admin bucket stats --bucket redacted { "bucket": "redacted", ........... "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 18446744073709551607 }, "rgw.main": { "size": 1687971465874, "size_actual": 1696692400128, "size_utilized": 1687971465874, "size_kb": 1648409635, "size_kb_actual": 1656926172, "size_kb_utilized": 1648409635, "num_objects": 4290147 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 75 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } ``` We did find a few reports on this issue, eg. http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-November/037531.ht…. Do we know any use patterns that can lead object count to become that large? Also is there a way to accurately collect the object count for each bucket in the cluster, as we would like to use it for management purpose.

1 0

Re: Is there a better way to make a samba/nfs gateway? (Marc Roos)
by Chad William Seys 14 Mar '20

14 Mar '20

Awhile back I thought there were some limitations which prevented us from trying this, but I cannot remember... What does the ceph vfs gain you over exporting by cephfs kernel module (kernel 4.19). What does it lose you? (I.e. pros and cons versus kernel module?) Thanks! C. > It's based on vfs_ceph and you can read more about how to configure it > yourself on > https://www.samba.org/samba/docs/current/man-html/vfs_ceph.8.html.

2 1

Re: Is there a better way to make a samba/nfs gateway?
by Konstantin Shalygin 14 Mar '20

14 Mar '20

On 3/13/20 10:47 PM, Chip Cox wrote: > Konstantin - in your Windows environment, would it be beneficial to > have the ability to have NTFS data land as S3 (object store) on a Ceph > storage appliance? Or does it have to be NFS? > > Thanks and look forward to hearing back. Nope, for windows we use CephFS over Samba VFS CTDB cluster. k

1 0