- ceph-users - lists.ceph.io

by Luca Cervigni

Hello, I have a two zones cluster. Default zone + a ES sync zone where I would like to index metadata and do searches via the ES tier gateway. Struggling to get this service running (documentation is bad and very, very minimal) I thought that in the end I made it work. The requests are getting correctly to the ES zone, I can see the payload sent to ES with debug rgw = 20, ES returns OK then an error is trigger, apparently a JSON parsing error taking the ES search and giving it back to the user. I tried with Nautilus, Octopus, and different versions of ES, from 6 to 7. I have no possibilities of downgrades as our production cluster is on nautilus. Here the logs: https://pastebin.com/hnj8YysK I would think this is a bug, but would like your opinion. If not a bug how shall I solve this? I am using a handmade S3 tool to query as boto is not supported anymore, obo just does not work, and boto3 cannot send any arbitrary requests anymore. Also I would like to know if those Sync modules are still supported or not. The documentation is very poor and if they still are in the roadmap, it should be improved. Thanks Cheers, Luca Cervigni Pawsey Supercomputing centre

3 years, 11 months

1
0
0 0

RGW crashed

by Zhenshi Zhou

Hi all, I deployed a multi-site cluster in order to sync object from an old cluster to a brand new cluster. It seems good cause I can see the data syncing. However, when I check the cluster health, it shows warn messages "2 daemons have recently crashed". I get the crash info by 'sudo ceph crash info $id': { "os_version_id": "7", "utsname_release": "3.10.0-957.27.2.el7.x86_64", "os_name": "CentOS Linux", "entity_name": "client.rgw.ceph-node7", "timestamp": "2020-05-09 15:17:59.482502Z", "process_name": "radosgw", "utsname_machine": "x86_64", "utsname_sysname": "Linux", "os_version": "7 (Core)", "os_id": "centos", "utsname_version": "#1 SMP Mon Jul 29 17:46:05 UTC 2019", "backtrace": [ "(()+0xf5f0) [0x7f32b1bdf5f0]", "(RGWCoroutine::set_sleeping(bool)+0xc) [0x555eeb1351ac]", "(RGWOmapAppend::flush_pending()+0x2d) [0x555eeb13acad]", "(RGWOmapAppend::finish()+0x10) [0x555eeb13acd0]", "(RGWDataSyncShardCR::stop_spawned_services()+0x2b) [0x555eeb0a185b]", "(RGWDataSyncShardCR::incremental_sync()+0x72a) [0x555eeb0a9baa]", "(RGWDataSyncShardCR::operate()+0x9d) [0x555eeb0ab33d]", "(RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x60) [0x555eeb136520]", "(RGWCoroutinesManager::run(std::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x236) [0x555eeb137196]", "(RGWCoroutinesManager::run(RGWCoroutine*)+0x78) [0x555eeb138098]", "(RGWRemoteDataLog::run_sync(int)+0x1cf) [0x555eeb08851f]", "(RGWDataSyncProcessorThread::process()+0x46) [0x555eeb1e71a6]", "(RGWRadosThread::Worker::entry()+0x115) [0x555eeb1b6195]", "(()+0x7e65) [0x7f32b1bd7e65]", "(clone()+0x6d) [0x7f32b10e188d]" ], "utsname_hostname": "ceph-node7", "crash_id": "2020-05-09_15:17:59.482502Z_b80d7bee-faa0-4d2f-9d86-a1b3f4d4802e", "ceph_version": "14.2.8" } AND { "os_version_id": "7", "utsname_release": "3.10.0-957.27.2.el7.x86_64", "os_name": "CentOS Linux", "entity_name": "client.rgw.ceph-node7", "timestamp": "2020-05-10 16:23:13.375063Z", "process_name": "radosgw", "utsname_machine": "x86_64", "utsname_sysname": "Linux", "os_version": "7 (Core)", "os_id": "centos", "utsname_version": "#1 SMP Mon Jul 29 17:46:05 UTC 2019", "backtrace": [ "(()+0xf5f0) [0x7f409f42e5f0]", "(RGWCoroutine::set_sleeping(bool)+0xc) [0x55e3f45e01ac]", "(RGWOmapAppend::flush_pending()+0x2d) [0x55e3f45e5cad]", "(RGWOmapAppend::finish()+0x10) [0x55e3f45e5cd0]", "(RGWDataSyncShardCR::stop_spawned_services()+0x2b) [0x55e3f454c85b]", "(RGWDataSyncShardCR::incremental_sync()+0x72a) [0x55e3f4554baa]", "(RGWDataSyncShardCR::operate()+0x9d) [0x55e3f455633d]", "(RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x60) [0x55e3f45e1520]", "(RGWCoroutinesManager::run(std::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x236) [0x55e3f45e2196]", "(RGWCoroutinesManager::run(RGWCoroutine*)+0x78) [0x55e3f45e3098]", "(RGWRemoteDataLog::run_sync(int)+0x1cf) [0x55e3f453351f]", "(RGWDataSyncProcessorThread::process()+0x46) [0x55e3f46921a6]", "(RGWRadosThread::Worker::entry()+0x115) [0x55e3f4661195]", "(()+0x7e65) [0x7f409f426e65]", "(clone()+0x6d) [0x7f409e93088d]" ], "utsname_hostname": "ceph-node7", "crash_id": "2020-05-10_16:23:13.375063Z_9e70a0c0-929e-445f-b4cd-8d29e909fe2f", "ceph_version": "14.2.8" } So I fetch and check the file "ceph-client.rgw.ceph-node7.log". The log has huge amount of errors like: -732> 2020-05-09 23:17:53.476 7f328b7ff700 0 RGW-SYNC:data:sync:shard[98]:entry[harbor-registry:f70a5eb9-d88d-42fd-ab4e-d300e97094de.4620.94:23]:bucket[harbor-registry:f70a5eb9-d88d-42fd-ab4e-d300e97094de.4620.94:23]:inc_sync[harbor-registry:f70a5 eb9-d88d-42fd-ab4e-d300e97094de.4620.94:23]: ERROR: lease is not taken, abort AND -723> 2020-05-09 23:17:56.388 7f328b7ff700 5 RGW-SYNC:data:sync:shard[88]:entry[harbor-registry:f70a5eb9-d88d-42fd-ab4e-d300e97094de.4620.94:13]:bucket[harbor-registry:f70a5eb9-d88d-42fd-ab4e-d300e97094de.4620.94:13]: incremental sync on bucket fa iled, retcode=-125 AND -215> 2020-05-09 23:17:58.809 7f328b7ff700 5 RGW-SYNC:data:sync:shard[10]:entry[pf2-harbor-swift:f70a5eb9-d88d-42fd-ab4e-d300e97094de.4608.101:113]:bucket[pf2-harbor-swift:f70a5eb9-d88d-42fd-ab4e-d300e97094de.4608.101:113]: full sync on bucket failed, retcode=-125 AND 2020-05-09 23:18:24.048 7f4085867700 1 robust_notify: If at first you don't succeed: (110) Connection timed out 2020-05-09 23:18:24.048 7f4083863700 0 ERROR: failed to distribute cache for shubei.rgw.log:datalog.sync-status.shard.f70a5eb9-d88d-42fd-ab4e-d300e97094de.5 2020-05-09 23:28:49.181 7f407e859700 1 heartbeat_map reset_timeout 'RGWAsyncRadosProcessor::m_tp thread 0x7f407e859700' had timed out after 600 2020-05-10 03:12:01.905 7f409708a700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror And finally it crashed. I'm not sure where the problem is. Were the crashes caused by the network? Thanks

3 years, 11 months

1
0
0 0

octopus cluster deploy with cephadm failed on bootstrap

by Zhenshi Zhou

Hi all, I'm deploying a new octopus cluster using cephadm, follow docs <https://ceph.io/ceph-management/introducing-cephadm/>. However it failed on the bootstrap step. According to the logs, key generating failed because of the lack of directory and file. Did I miss something? here is the logs: [root@ceph-mon1 ~]# cephadm bootstrap --mon-ip 172.24.202.119 INFO:cephadm:Verifying podman|docker is present... INFO:cephadm:Verifying lvm2 is present... INFO:cephadm:Verifying time synchronization is in place... INFO:cephadm:Unit ntpd.service is enabled and running INFO:cephadm:Repeating the final host check... INFO:cephadm:podman|docker (/usr/bin/docker) is present INFO:cephadm:systemctl is present INFO:cephadm:lvcreate is present INFO:cephadm:Unit ntpd.service is enabled and running INFO:cephadm:Host looks OK INFO:root:Cluster fsid: 828e6b7a-91c2-11ea-8644-005056885573 INFO:cephadm:Verifying IP 172.24.202.119 port 3300 ... INFO:cephadm:Verifying IP 172.24.202.119 port 6789 ... INFO:cephadm:Mon IP 172.24.202.119 is in CIDR network 172.24.202.0/24 INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container... INFO:cephadm:Extracting ceph user uid/gid from container image... INFO:cephadm:Creating initial keys... INFO:cephadm:Creating initial monmap... INFO:cephadm:Creating mon... INFO:cephadm:Waiting for mon to start... INFO:cephadm:Waiting for mon... INFO:cephadm:Assimilating anything we can from ceph.conf... INFO:cephadm:Generating new minimal ceph.conf... INFO:cephadm:Restarting the monitor... INFO:cephadm:Setting mon public_network... INFO:cephadm:Creating mgr... INFO:cephadm:Wrote keyring to /etc/ceph/ceph.client.admin.keyring INFO:cephadm:Wrote config to /etc/ceph/ceph.conf INFO:cephadm:Waiting for mgr to start... INFO:cephadm:Waiting for mgr... INFO:cephadm:mgr not available, waiting (1/10)... INFO:cephadm:mgr not available, waiting (2/10)... INFO:cephadm:mgr not available, waiting (3/10)... INFO:cephadm:mgr not available, waiting (4/10)... INFO:cephadm:mgr not available, waiting (5/10)... INFO:cephadm:Enabling cephadm module... INFO:cephadm:Waiting for the mgr to restart... INFO:cephadm:Waiting for Mgr epoch 4... INFO:cephadm:Setting orchestrator backend to cephadm... INFO:cephadm:Generating ssh key... INFO:cephadm:Non-zero exit code 22 from /usr/bin/docker run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=ceph-mon1 -v /var/log/ceph/828e6b7a-91c2-11ea-8644-005056885573:/var/log/ceph:z -v /tmp/ceph-tmp8ydndiho:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp9yl7hx5u:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15 cephadm generate-key INFO:cephadm:/usr/bin/ceph:stderr Error EINVAL: Traceback (most recent call last): INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/cephadm/module.py", line 1438, in _generate_key INFO:cephadm:/usr/bin/ceph:stderr with open(path, 'r') as f: INFO:cephadm:/usr/bin/ceph:stderr FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpxvc18jh3/key' INFO:cephadm:/usr/bin/ceph:stderr INFO:cephadm:/usr/bin/ceph:stderr During handling of the above exception, another exception occurred: INFO:cephadm:/usr/bin/ceph:stderr INFO:cephadm:/usr/bin/ceph:stderr Traceback (most recent call last): INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command INFO:cephadm:/usr/bin/ceph:stderr return self.handle_command(inbuf, cmd) INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 110, in handle_command INFO:cephadm:/usr/bin/ceph:stderr return dispatch[cmd['prefix']].call(self, cmd, inbuf) INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/mgr_module.py", line 308, in call INFO:cephadm:/usr/bin/ceph:stderr return self.func(mgr, **kwargs) INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 72, in <lambda> INFO:cephadm:/usr/bin/ceph:stderr wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 63, in wrapper INFO:cephadm:/usr/bin/ceph:stderr return func(*args, **kwargs) INFO:cephadm:/usr/bin/ceph:stderr File "/usr/share/ceph/mgr/cephadm/module.py", line 1443, in _generate_key INFO:cephadm:/usr/bin/ceph:stderr os.unlink(path) INFO:cephadm:/usr/bin/ceph:stderr FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpxvc18jh3/key' INFO:cephadm:/usr/bin/ceph:stderr Traceback (most recent call last): File "./cephadm", line 4579, in <module> r = args.func() File "./cephadm", line 1122, in _default_image return func() File "./cephadm", line 2521, in command_bootstrap cli(['cephadm', 'generate-key']) File "./cephadm", line 2409, in cli ).run(timeout=timeout) File "./cephadm", line 2142, in run self.run_cmd(), desc=self.entrypoint, timeout=timeout) File "./cephadm", line 837, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/docker run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=ceph-mon1 -v /var/log/ceph/828e6b7a-91c2-11ea-8644-005056885573:/var/log/ceph:z -v /tmp/ceph-tmp8ydndiho:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp9yl7hx5u:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15 cephadm generate-key OS: CentOS Linux release 7.8.2003 (Core) Kernel: 3.10.0-514.6.1.el7.x86_64 Docker version: 19.03.8 cephadm: Using recent ceph image ceph/ceph:v15 ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable) Thanks

3 years, 11 months

1
1
0 0

Cluster rename procedure

by Anthony D'Atri

I’ve inherited a couple of clusters with non-default (ie, not “ceph”) internal names, and I want to rename them for the usual reasons. I had previously developed a full list of steps - which I no longer have access to. Anyone done this recently? Want to be sure I’m not missing something. * Nautilus, CentOS 7, RGW and RBD * Rename OSD mountpoints with mount —move * Rename systemd resources / mounts? * Rename /var/lib/ceph/{mon,osd} directories * Rename ceph*conf files on backend and client systems * Rename keyrings — just the filenames? * Rename log files * Ajust `ceph config` paths for admin socket, keyring, logs, mgr/mds/mon data, osd journal, rgw_data * Restart daemons * Ensure /var/run/ceph sockets are appropriately named Thanks — aad

3 years, 12 months

2
1
0 0

Nautilus cluster rados gateway not sharding bucket indexes

by Marcel Kuiper

Hi Sorry for the repost, but I didn't get any respons on my first post so I will try to rephrase it We have several ceph clusters running nautilus (coming from mimic). On one of the clusters I got a health warning HEALTH_WARN 1 large omap objects * when I check on the rados gateway with radosgw-admin bucket limit check' I see 2 buckets over the limit * the config value for rgw_dynamic_resharding is true * on: 'radosgw-admin reshard stale-instances list' I get an error: Resharding disabled in a multisite env, stale instances unlikely from resharding. This is however a single site cluster. I have more clusters that are exactly the same and that do auto index resharding just fine. ANd above command does not give an error Can anyone help me tackle this problem. I've gone through the documentation and some blogs but did not find a solution yet Any help is much appreciated Thanks Marcel

3 years, 12 months

2
2
0 0

Re: Data loss by adding 2OSD causing Long heartbeat ping times

by Frank Schilder

On all OSD nodes I'm using vm.min_free_kbytes = 4194304 (4GB). This was one of the first tunings on the cluster. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri(a)gmail.com> Sent: 08 May 2020 10:17 To: Frank Schilder Subject: Re: [ceph-users] Re: Data loss by adding 2OSD causing Long heartbeat ping times Just for grins, what is your vm.min_free_kbytes setting? Depending on the size and number of your OSDs and your workload, 2-4GB is a starting point. > > Hi XuYun and Martin, > > I checked that already. The OSDs in question have 8GB memory limit and the RAM of the servers is about 50% used. It could be memory fragmentation, which used to be a problem before bitmap allocator. However, my OSDs are configured to use bitmap, at least that is what they claim they are using. > > There might be a somewhat more fundamental issue, also related to my recent experience described in "Ceph meltdown, need help". The problems seem to have the same source, busy OSDs get behind with their internal cluster communication because (I suspect) client IO and admin IO (heartbeats, beacons, etc.) are handled in the same queue. If things get a bit busy, admin I/O slows down and avalanches happen. > > Also in the case here (long heartbeat) I first saw remapping and peering going on, then the heartbeats times of a few OSDs suddenly shot up. It is possible that some OSDs were already busy with client I/O. The additional peering seems to have the ability to add so much additional load to some OSDs that they start falling behind and getting marked out erroneously, with the consequence of even more peering, load, etc. > > I'm working on a new conversation "Cluster outage due to client IO" to have a clean focused thread. I need a bit more time to collect information though. For now, our cluster is up and running healthy. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Martin Verges <martin.verges(a)croit.io> > Sent: 07 May 2020 12:17:10 > To: XuYun > Cc: Frank Schilder; ceph-users > Subject: Re: [ceph-users] Re: Data loss by adding 2OSD causing Long heartbeat ping times > > Hello XuYun, > > In my experience, I would always disable swap, it won't do any good. > > -- > Martin Verges > Managing director > > Mobile: +49 174 9335695 > E-Mail: martin.verges(a)croit.io<mailto:martin.verges@croit.io> > Chat: https://t.me/MartinVerges > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > > Web: https://croit.io > YouTube: https://goo.gl/PGE1Bx > > > Am Do., 7. Mai 2020 um 12:07 Uhr schrieb XuYun <yunxu(a)me.com<mailto:yunxu@me.com>>: > We had got some ping back/front problems after upgrading from filestore to bluestore. It turned out to be related to insufficient memory/swap usage. > >> 2020年5月6日下午10:08，Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> 写道： >> >> To answer some of my own questions: >> >> 1) Setting >> >> ceph osd set noout >> ceph osd set nodown >> ceph osd set norebalance >> >> before restart/re-deployment did not harm. I don't know if it helped, because I didn't retry the procedure that led to OSDs going down. See also point 3 below. >> >> 2) A peculiarity of this specific deployment of 2 OSDs was, that it was a mix of OSD deployment and restart after a reboot. I'm working on getting this sorted and this is a different story. For anyone who might find him-/herself in a situation where some OSDs are temporarily down/out with PGs remapped and objects degraded for whatever reason while new OSDs come up, the way to have ceph rescan the down/out OSDs after they come up is to >> >> - "ceph osd crush move" the new OSDs temporarily to a location outside the crush sub tree covering any pools (I have such a parking space in the crush hierarchy for easy draining and parking disks) >> - bring up the down/out OSDs >> - at this point, the cluster will fall back to the original crush map that was in place when the OSDs went down/out >> - the cluster will now find all shards that went orphan and health will be restored very quickly >> - once the cluster is healthy, "ceph osd crush move" the new OSDs back to their desired location >> - now you will see remapped PGs/misplaced objects, but no degraded objects >> >> 3) I still don't have an answer why long heartbeat ping times were observed. There seems to be a more serious issue and this will continue in its own thread "Cluster outage due to client IO" to be opened soon. >> >> Best regards, >> ================= >> Frank Schilder >> AIT Risø Campus >> Bygning 109, rum S14 >> >> ________________________________________ >> From: Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> >> Sent: 25 April 2020 15:34:25 >> To: ceph-users >> Subject: [ceph-users] Data loss by adding 2OSD causing Long heartbeat ping times >> >> Dear all, >> >> Two days ago I added very few disks to a ceph cluster and run into a problem I have never seen before when doing that. The entire cluster was deployed with mimic 13.2.2 and recently upgraded to 13.2.8. This is the first time I added OSDs under 13.2.8. >> >> I had a few hosts that I needed to add 1 or 2 OSDs to and I started with one that needed 1. Procedure was as usual: >> >> ceph osd set norebalance >> deploy additional OSD >> >> The OSD came up and PGs started peering, so far so good. To my surprise, however, I started seeing health-warnings about slow ping times: >> >> Long heartbeat ping times on back interface seen, longest is 1171.910 msec >> Long heartbeat ping times on front interface seen, longest is 1180.764 msec >> >> After peering it looked like it got better and I waited it out until the messages were gone. This took a really long time, at least 5-10 minutes. >> >> I went on to the next host and deployed 2 new OSDs this time. Same as above, but with much worse consequences. Apparently, the ping times exceeded a timeout for a very short moment and an OSD was marked out for ca. 2 seconds. Now all hell broke loose. I got health errors with the dreaded "backfill_toofull", undersized PGs and a large amount of degraded objects. I don't know what is causing what, but I ended up with data loss by just adding 2 disks. >> >> We have dedicated network hardware and each of the OSD hosts has 20GBit front and 40GBit back network capacity (LACP trunking). There are currently no more than 16 disks per server. The disks were added to an SSD pool. There was no traffic nor any other exceptional load on the system. I have ganglia resource monitoring on all nodes and cannot see a single curve going up. Network, CPU utilisation, load, everything below measurement accuracy. The hosts and network are quite overpowered and dimensioned to host many more OSDs (in future expansions). >> >> I have three questions, ordered by how urgently I need an answer: >> >> 1) I need to add more disks next week and need a workaround. Will something like this help avoiding the heartbeat time-out: >> >> ceph osd set noout >> ceph osd set nodown >> ceph osd set norebalance >> >> 2) The "lost" shards of the degraded objects were obviously still on the cluster somewhere. Is there any way to force the cluster to rescan OSDs for the shards that went orphan during the incident? >> >> 3) This smells a bit like a bug that requires attention. I was probably just lucky that I only lost 1 shard per PG. Has something similar reported before? Is this fixed in 13.2.10? Is it something new? Any settings that need to be looked at? If logs need to be collected, I can do so during my next attempt. However, I cannot risk data integrity of a production cluster and, therefore, probably not run the original procedure again. >> >> Many thanks for your help and best regards, >> ================= >> Frank Schilder >> AIT Risø Campus >> Bygning 109, rum S14 >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 years, 12 months

1
0
0 0

ceph-mgr high CPU utilization

by Andras Pataki

I'm wondering if anyone still sees issues with ceph-mgr using CPU and being unresponsive even in recent Nautilus releases. We upgraded our largest cluster from Mimic to Nautilus (14.2.8) recently - it has about 3500 OSDs. Now ceph-mgr is constantly at 100-200% CPU (1-2 cores), and becomes unresponsive after a few minutes. The finisher-Mgr queue length grows (I've seen it at over 100k) - similar symptoms as seen with earlier Nautilus releases by many. This is what it looks like after an hour of running: "finisher-Mgr": { "queue_len": 66078, "complete_latency": { "avgcount": 21, "sum": 2098.408767721, "avgtime": 99.924227034 } }, We have a pretty vanilla manager config, only the balancer is enabled in upmap mode. Here are the enabled modules: "always_on_modules": [ "balancer", "crash", "devicehealth", "orchestrator_cli", "progress", "rbd_support", "status", "volumes" ], "enabled_modules": [ "restful" ], Any ideas or outstanding issues in this area? Andras

3 years, 12 months

3
6
0 0

Unit testing of CRUSH Algorithm

by Bobby

Hi, Are there any more resources of unit tests for CRUSH algorithm other than the test cases here: : https://github.com/ceph/ceph/tree/master/src/test/crush Or more unit testing of CRUSH apart from the these test cases would be an overkill? BR Bobby !

3 years, 12 months

1
0
0 0

Cluster network and public network

by Nghia Viet Tran

Hi everyone, I have a question about the network setup. From the document, It’s recommended to have 2 NICs per hosts as described in below picture [Diagram] In the picture, OSD hosts will connect to the Cluster network for replicate and heartbeat between OSDs, therefore, we definitely need 2 NICs for it. But seems there are no connections between Ceph MON and Cluster network. Can we install 1 NIC on Ceph MON then? I appreciated any comments! Thank you! -- Nghia Viet Tran (Mr)

3 years, 12 months

2
1
0 0

cephfs change/migrate default data pool

by Kenneth Waegeman

Hi all, I read in some release notes it is recommended to have your default data pool replicated and use erasure coded pools as additional pools through layouts. We have still a cephfs with +-1PB usage with a EC default pool. Is there a way to change the default pool or some other kind of migration without having to recreate the FS? Thanks! Kenneth

3 years, 12 months

3
4
0 0

2024

2023

2022

2021

2020

2019

ceph-users