- ceph-users - lists.ceph.io

Migrated to cephadm, rgw logs to file even when rgw_ops_log_rados is true
by Paul JURCO 28 Nov '24

28 Nov '24

Hi, we recently migrated to cephadm from ceph-deploy a 18.2.2 ceph cluster (Ubuntu with docker). RGWs are separate vms. We noticed syslog increased a lot due to rgw's access logs sent to it. And because we use to log ops, a huge ops log file on /var/log/ceph/cluster-id/ops-log-ceph-client.rgw.hostname-here.log. While having "rgw_ops_log_rados": "true", oplogs goes to both file and rados pool for logs. If false it doesn't log anything, as expected. How to stop dockered rgws to log to syslog and to a file on disk, but to keep opslog in logs pool? Config is: global basic log_to_journald false global advanced rgw_enable_ops_log false global advanced rgw_ops_log_rados true A few hours later after after enabling it back, after massive cleanup, it does logging ops but only to files. How to get ops logs in rados pool and access log to a file on disk but not on syslog? I have add this to daemon.json to limit access logs to accumulate on /var/log/docker/containers/rand/rand/json.log file: { "log-driver": "local", "log-opts": { "max-size": "512m", "max-file": "3" } } Thank you! Paul

2 3

testing with tcmu-runner vs rbd map
by Marc 27 Nov '24

27 Nov '24

I know this is not a good test, but when I dd to a rbd image like this [@ ~]# dd if=/dev/zero of=/dev/rbd0 bs=1M count=100 status=progress 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.399629 s, 262 MB/s It is a big cache difference doing this via the tcmu-runner [@~]# dd if=/dev/zero of=/dev/sdac bs=1M count=100 status=progress 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3571 s, 10.1 MB/s Is there anything I can change so the default config of tcmu-runner behaves like the rbdmap?

1 0

Ceph Nautilus packages for ubuntu 20.04
by Pardhiv Karri 27 Nov '24

27 Nov '24

Hi, I am in a tricky situation. Our current OSD nodes (luminous version) are on the latest Dell servers, which only support Ubuntu 20.04. The Luminous packages were installed on 16.04, so the packages are still Xenial; I later upgraded the OS to 20.04 and added OSDs to the cluster. Now, I am trying to upgrade to Nautilus while on Ubuntu 20.04, but there are no Nautilus packages for 20.04. What is the best way to upgrade OSD nodes to Nautilus on 20.04? The Mon nodes have already been upgraded to Nautilus, as they are still on Ubuntu 16.04 (older Dell nodes). Thanks, Pardhiv

3 2

ceph cluster planning size / disks
by Marc 27 Nov '24

27 Nov '24

I was wondering if there is some online tool that can help you with calculating usable storage from nodes, drives per node, replication/erasure used etc.

7 10

Balancer: Unable to find further optimization
by sinan＠turka.nl 27 Nov '24

27 Nov '24

Hi, My Ceph cluster is out-of-balance. The amount of PG's per OSD ranges from about 50 up to 100 PG's per OSD. This is far from balanced. Today, I have enabled the balancer module. But unfortunately, it doesn't want to balance: "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect" The upmap_max_deviation setting was set to default (5). Changing this to 1, triggered the balancer module. It did 1 run, balanced a few PG's (less than 5 PG's in total), then again it got stuck with the same message: "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect" I have about 280 OSD's: 24 OSD's per node, 4 nodes per rack, 3 racks in total. Replica = 3. 1 replica per rack. My disk sizes differs from 1.6T up to 2.4T. Any advice on this? Thanks! Sinan

2 4

iscsi-ceph
by Marc 27 Nov '24

27 Nov '24

Failed: Clients can not be defined until a HA configuration has been defined (>2 gateways) Who cares when I am testing???? I am fully aware I only entered 1.

1 0

iscsi testing
by Marc 27 Nov '24

27 Nov '24

what is the best location to get tcmu-runner rpms from? (Does not seem to be available in el9) create ceph-gw-1 10.172.19.21 skipchecks=true returns: The first gateway defined must be the local machine I can only put here full domainname not even only the hostname, does not seem to match the manual.

1 0

macos rbd client
by Marc 27 Nov '24

27 Nov '24

Does it make sense to try and see if I can connect macos client to an rbd device, or is this never going to be a stable supported environment? Are people doing this?

1 0

CephFS 16.2.10 problem
by Alexey.Tsivinsky＠baikalelectronics.ru 27 Nov '24

27 Nov '24

Good evening! The following problem occurred. There is a cluster ceph 16.2.10 The cluster was operating normally on Friday. Shut down cluster: -Excluded all clients Executed commands: ceph osd set noout ceph osd set nobackfill ceph osd set norecover ceph osd set norebalance ceph osd set nodown ceph osd set pause Turned off the cluster, checked server maintenance. Enabled cluster. He gathered himself, found all the nodes, and here the problem began. After all OSD went up and all pg became available, cephfs refused to start. Now mds are in the replay status, and do not go to the ready status. Previously, one of them was in the replay (laggy) status, but we executed command: ceph config set mds mds_wipe_sessions true After that, mds switched to the status of replays, the third in standby status started, and mds crashes with an error stopped. But cephfs is still unavailable. What else can we do? The cluster is very large, almost 200 million files. Best regards A.Tsivinsky e-mail: alexey.tsivinsky(a)baikalelectronics.com

3 8

config set -> ceph.conf
by Marc 27 Nov '24

27 Nov '24

How should I rewrite this to ceph.conf ceph config set mon mon_warn_on_insecure_global_id_reclaim false ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false

2 1