- ceph-users - lists.ceph.io

OSD and RBD on same node?
by Tuffli, Chuck 25 May '21

25 May '21

I found a warning in Red Hat documentation regarding OSD and RBD: Ceph Block Devices must be deployed on separate nodes from the Ceph Monitor and OSD nodes. Running kernel clients and kernel server daemons on the same node can lead to kernel deadlocks. [1] It was hard to tell from the documentation if this is still true as some Rook documentation [2] mentioned a solution. Is having an RBD on the same node as the OSD's still a no-no or is it OK with the conditions listed in the Rook docs (i.e. patched kernel + no XFS)? TIA! --chuck [1] https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/i… [2] https://rook.io/docs/rook/v1.6/ceph-common-issues.html#a-worker-node-using-…

3 2

DocuBetter Meeting 1AM UTC Thursday 27 May 2021
by John Zachary Dover 24 May '21

24 May '21

A DocuBetter meeting is scheduled for later this week at 11AM AEST Thursday, which is 6PM PDT Wednesday and 9PM EDT Wednesday (and 1AM UTC). It's a pretty inconveniently scheduled meeting. Last month I sent an email reporting that this monthly meeting, on account of its inconvenient time, is not much attended. That email provoked good, helpful correspondence, so I'm doing it again. Unless I get responses to this email thread that request that I hold the meeting this week, I'm not going to hold it. This email is a sincere request for documentation complaints. If anything about the documentation irritates you, now's the time to tell me. If anything about the documentation is incomplete or incorrect, I'm the guy you should tell. You don't have to attend the DocuBetter Meetings to get changes into the documentation, you just have to ask me and explain clearly what needs to be changed. The documentation initiatives that were underway last month remain underway sedulously, if not apace. I am cleaning the syntax of the entire documentation suite (a multi-month project that is as tedious as it sounds... I am currently in the middle of the cephadm documentation). I am also planning to consolidate the Intro Guide and part of the Developer Guide, because there is some overlap in those sections. I've also begun indexing the Ceph-related videos on Youtube. This means that I watch the videos with a pencil in my hand, and I write down short summaries of what's being discussed along with the number of minutes and seconds that have passed since the video began. Over the past two weeks I have found myself with command of the context and understanding necessary to appreciate Sage's one-hour-and-twenty-seven-minute marathon "Intro to Ceph" talk from June 2019 (https://www.youtube.com/watch?v=PmLPbrf-x9g) I'm not sure that a video six minutes longer than "Toy Story" can be considered an intro, but it is an impressive rhetorical performance by Sage, who speaks authoritatively and concisely on every subject he touches as he provides a tour of all the major components of Ceph. Anyway: If someone responds to this email and says that their lives would be improved if I hold the DocuBetter meeting, then I will hold the meeting. However, if by twelve hours before the time of the meeting no one has responded, there will be no DocuBetter meeting. Remember: even if no one wants to have the meeting this week, this does not mean that you can't get changes into the docs. Write to me anytime with your complaints. No complaint is too crude, no irritation will be summarily dismissed. That's it. Here are the links relevant to DocuBetter Meetings: Meeting: https://bluejeans.com/908675367 Etherpad: https://pad.ceph.com/p/Ceph_Documentation

1 0

upmap+assimilate-conf clarification
by Szabo, Istvan (Agoda) 24 May '21

24 May '21

Hi, I've successfully updated my luminous lab environment to nautilus so next week I'll give a try to the prod env but 2 things same up in the description: 1. Upmap: I've never used this before don't know how I couldn't see it because it is quite cool feature. So ceph says let it in upmap, my question is like with the ceph-pg autoscaler module not the best thing to use in 'on' mode, better in warn, so is it good to use in upmap mode let's say in ceph world "safe"? 2. Regarding this assimilate-conf. It's not clear for me what is it actually? Doc says: "This is also a good time to fully transition any config options in ceph.conf into the cluster's configuration database. On each host, you can use the following command to import any option into the monitors with ceph config assimilate-conf -i /etc/ceph/ceph.conf". What does this mean? If I have different configs on different servers and I run it on the servers it will merge it together all the configs? Or what I can do with this feature? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Does dynamic resharding block I/Os by design?
by Satoru Takeuchi 24 May '21

24 May '21

Hi, I have a Ceph cluster used for RGW and RBD. I found that all I/Os to RGW seemed to be blocked while dynamic resharding. Could you tell me whether this behavior is by design or not? I attached a graph which means I/O seemed to be blocked. Here x-axis is time and y-axis is the number of RADOS objects. In addition, dynamic resharding was run between 16:22:30 and 16:31:30. I read the official documents about dynamic resharding. But there is no description about blocking during dynamic resharding. https://docs.ceph.com/en/octopus/radosgw/dynamicresharding/ In addition, I read the following Red Hat's blog post. https://www.redhat.com/ja/blog/ceph-rgw-dynamic-bucket-sharding-performance… > You do not need to stop reading or writing objects to the bucket while resharding is happening. It would mean the dynamic resharding is online operation. However, it's not clear whether this feature blocks I/Os or not. Thanks, Satoru

2 4

How to organize data in S3
by Michal Strnad 24 May '21

24 May '21

Hi all, We need to store millions of files using S3 protocol in Ceph (version Nautilus), but have projects where isn't appropriate or possible to create a lot of S3 accounts. Is it better to have multiple S3 buckets or one bucket with sub folders? For example AWS service from Amazon allows you to create up to 100 buckets in each of your AWS cloud accounts. You can request more buckets, up to a maximum quota of 1,000, by submitting a service limit increase. There is no limit on the number of objects you can store in a bucket, but in the Ceph we run into a problem with listing and resharding with a millions of files in one bucket. Thank you Michal

4 5

Recommendations on problem with PG
by Gabriel Medve 23 May '21

23 May '21

Hi, We have a problem with a PG that was inconsistent, currently the PG in our cluster have 3 copies. It was not possible for us to repair this pg with "ceph pg repair" (This PG is in osd 14,1,2) so we deleted some of the copies of osd 14 with the following command. ceph-objectstore-tool --data-path /var/lib/ceph/osd.14/ --pgid 22.f --op remove --force This caused an automatic attempt to create the missing copy entering the backfilling state, but when doing this it crashed osd 1 and 2 and threw the IOPS to 0, freezing the cluster. Is there any way to remove this entire pg or try to recreate the missing copy or ignore it completely? It causes instability in the cluster. Thank you, I await comments -- Untitled Document ------------------------------------------------------------------------ Gabriel I. Medve

2 2

One mds daemon damaged, filesystem is offline. How to recover?
by Sagara Wijetunga 23 May '21

23 May '21

Hi all An accidental power failure happened. That resulted CephFS offline and cannot be mounted. I have 3 MDS daemons but it complains "1 mds daemon damaged". It seems a PG of cephfs_metadata is inconsistent. I tried to repair, but doesn't get it repaired. How do I repair the damaged MDS and bring the CephFS up/online? Details are included below. Many thanks in advance. Sagara # ceph -s cluster: id: abc... health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 4 scrub errors Possible data damage: 1 pg inconsistent services: mon: 3 daemons, quorum a,b,c (age 107s) mgr: a(active, since 22m), standbys: b, c mds: cephfs:0/1 3 up:standby, 1 damaged osd: 3 osds: 3 up (since 96s), 3 in (since 96s) data: pools: 3 pools, 192 pgs objects: 281.05k objects, 327 GiB usage: 2.4 TiB used, 8.1 TiB / 11 TiB avail pgs: 191 active+clean 1 active+clean+inconsistent # ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged; 4 scrub errors; Possible data damage: 1 pg inconsistent FS_DEGRADED 1 filesystem is degraded fs cephfs is degraded MDS_ALL_DOWN 1 filesystem is offline fs cephfs is offline because no MDS is active for it. MDS_DAMAGE 1 mds daemon damaged fs cephfs mds.0 is damaged OSD_SCRUB_ERRORS 4 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 2.44 is active+clean+inconsistent, acting [0,2,1] # ceph osd lspools 2 cephfs_metadata 3 cephfs_data 4 rbd # ceph pg repair 2.44 # ceph -w 2021-05-22 01:48:04.775783 osd.0 [ERR] 2.44 shard 0 soid 2:22efaf6a:::200.00006048:head : candidate size 1540096 info size 1555896 mismatch 2021-05-22 01:48:04.775786 osd.0 [ERR] 2.44 shard 1 soid 2:22efaf6a:::200.00006048:head : candidate size 1540096 info size 1555896 mismatch 2021-05-22 01:48:04.775787 osd.0 [ERR] 2.44 shard 2 soid 2:22efaf6a:::200.00006048:head : candidate size 1441792 info size 1555896 mismatch 2021-05-22 01:48:04.775789 osd.0 [ERR] 2.44 soid 2:22efaf6a:::200.00006048:head : failed to pick suitable object info 2021-05-22 01:48:04.775849 osd.0 [ERR] repair 2.44 2:22efaf6a:::200.00006048:head : on disk size (1540096) does not match object info size (1555896) adjusted for ondisk to (1555896) 2021-05-22 01:48:04.787167 osd.0 [ERR] 2.44 repair 4 errors, 0 fixed --- End of detail ---

3 15

Re: One mds daemon damaged, filesystem is offline. How to recover?
by Eugen Block 22 May '21

22 May '21

Oh these are filestore OSDs? I didn't expect that. > 2021-05-20 23:56:52.223 2227200 0 mds.0.journaler.mdlog(ro) > _finish_read got less than expected (1555896) > 2021-05-20 23:56:52.223 2229180 0 mds.0.log _replay journaler got > error -22, aborting > 2021-05-20 23:56:52.223 2229180 -1 log_channel(cluster) log [ERR] : > Error loading MDS rank 0: (22) Invalid argument This confirms that the MDS is not starting because of the object size. I assume there was some recovery going on when this happened? The OSD uptime from your status was quite short. I'm not really sure what to do next, I found an old thread [4] with almost the same error, only that the user ended up truncating the objects because the expected size was smaller than the object info, in your case they're the other way around. In any case I would backup all three copies of that object including md5sum before chaning anything. [4] https://www.spinics.net/lists/ceph-devel/msg16516.html Zitat von Sagara Wijetunga <sagarawmw(a)yahoo.com>: > Here are the physical file sizes of the "200.00006048*": > OSD.0:-rw-r--r-- 1 ceph ceph 1540096 May 20 22:47 > /var/lib/ceph/osd/ceph-0/current/2.44_head/200.00006048__head_56F5F744__ > > OSD.1:-rw-r--r-- 1 ceph ceph 1540096 May 20 22:47 > /var/lib/ceph/osd/ceph-1/current/2.44_head/DIR_4/DIR_4/200.00006048__head_56F5F744__2 > > OSD2:-rw-r--r-- 1 ceph ceph 1441792 May 20 22:47 > /var/lib/ceph/osd/ceph-2/current/2.44_head/DIR_4/DIR_4/200.00006048__head_56F5F744__2 > > Sagara > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: ceph orch status hangs forever
by ManuParra 22 May '21

22 May '21

Hi Eugen, this is the output: ceph mgr module ls { "always_on_modules": [ "balancer", "crash", "devicehealth", "orchestrator", "pg_autoscaler", "progress", "rbd_support", "status", "telemetry", "volumes" ], "enabled_modules": [ "cephadm", "dashboard", "diskprediction_local", "iostat", "prometheus", "restful" ], "disabled_modules": [ ... } As you see balancer/crash/… are in section always_on. I checked it on all of the 3 monitor nodes with the same output. Then, checking disabled_modules I’ve seen many modules that could help to track some more information (logs) on our problem, like: - alerts - insights - test_orchestrator - and other… But we are not sure if we can enable some of them. Now all the logs we have from Ceph are not showing errors. Would it help to see more logs to enable some of those modules? On the other hand, as for what you indicate us of the commands that hang, we see that the containers that launch it remain executing, since they are waiting to finish. So you can see here the list of tests that remain running (hung): af13bda77a1a 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph osd status" 23 hours ago Up 23 hours wizardly_leavitt 5b5c760454c7 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph telemetry stat…" 24 hours ago Up 24 hours intelligent_bardeen a98e6061489d 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph service dump" 24 hours ago Up 24 hours romantic_mendel 66c943a032f8 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph service status" 24 hours ago Up 24 hours happy_shannon 7e18899dffc5 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph crash stat" 24 hours ago Up 24 hours xenodochial_germain 8268082e753b 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph crash ls" 24 hours ago Up 24 hours stoic_volhard fc5c434a4e23 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph balancer status" 24 hours ago Up 24 hours epic_mendel So the containers will have to be eliminated. As for the logs of these containers nothing appears inside the container (docker logs xxxx), only, when you kill it you can see (--verbose): [ceph: root@spsrc-mon-1 /]# ceph —verbose pg stat …. validate_command: pg stat better match: 2.5 > 0: pg stat bestcmds_sorted: [{'flags': 8, 'help': 'show placement group status.', 'module': 'pg', 'perm': 'r', 'sig': [argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, name=prefix, n=1, numseen=0, prefix=pg), argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, name=prefix, n=1, numseen=0, prefix=stat)]}] Submitting command: {'prefix': 'pg stat', 'target': ('mon-mgr', '')} submit ['{"prefix": "pg stat", "target": ["mon-mgr", ""]}'] to mon-mgr [hung forever …] Kind regards, Manu.

3 3

Force processing of num_strays in mds
by Mark Schouten 22 May '21

22 May '21

Hi, I have a 12.2.13 I want to go and upgrade. However, there are a whole bunch of stray files/inodes(?) which I would want to have processed. Also because I get a lot of 'No space left on device' messages. I started a 'find . -ls' in the root of the CephFS filesystem, but that causes overload and takes a lot of time, while not neccesarily fixing the num_strays. How do I force the mds'es to process those strays so that clients do not get 'incorrect' errors? -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | info(a)tuxis.nl

2 1