- ceph-users - lists.ceph.io

Re: Single machine / multiple monitors
by Brian Topping 12 Mar '20

12 Mar '20

If the ceph roadmap is getting rid of named clusters, how will multiple clusters be supported? How (for instance) would `/var/lib/ceph/mon/{name}` directories be resolved? > On Mar 11, 2020, at 8:29 PM, Brian Topping <brian.topping(a)gmail.com> wrote: > >> On Mar 11, 2020, at 7:59 PM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: >> >>> This is all possible with a single cluster, but this limited node also needs storage. >> >> Are you saying that the limited node needs to access Ceph-based storage? Is this some sort of converged architecture? > > It is a converged architecture in that all three boxes are running Kubernetes. There is one k8s cluster on each side of the link, let’s call them “primary” and “secondary”: > * The primary k8s cluster will only access storage from the primary Ceph cluster, secondary k8s only accesses storage from secondary Ceph. > * Primary Ceph gets monitors on both sides of the link. Secondary Ceph only has monitors on the secondary side. > > In a netsplit situation, the primary Ceph will maintain quorum with both nodes on the primary side. The secondary Ceph cluster only exists separately for this netsplit situation and the secondary k8s cluster can continue unaffected. > > With this in place, the primary side can continue operating with either primary node downed for maintenance via a suboptimal quorum over the WAN link. I cannot do that today. > > I am sacrificing the case where there is a netsplit at the same time I am doing maintenance. > > Thanks for your input! > Brian > >

1 0

RGWReshardLock::lock failed to acquire lock ret=-16
by Josh Haft 12 Mar '20

12 Mar '20

Hi, Currently running Mimic 13.2.5. We had reports this morning of timeouts and failures with PUT and GET requests to our Ceph RGW cluster. I found these messages in the RGW log: RGWReshardLock::lock failed to acquire lock on bucket_name:bucket_instance ret=-16 NOTICE: resharding operation on bucket index detected, blocking block_while_resharding ERROR: bucket is still resharding, please retry Which were preceded by many of these, which I think are normal/expected. check_bucket_shards: resharding needed: stats.num_objects=6415879 shard max_objects=6400000 Our RGW cluster sits behind haproxy which notified me approx 90 seconds after the first 'resharding needed' message that no backends were available. It appears this dynamic reshard process caused the RGWs to lock up for a period of time. Roughly 2 minutes later the reshard error messages stop and operation returns to normal. Looking back through previous RGW logs, I see a similar event from about a week ago, on the same bucket. We have several buckets with shard counts exceeding 1k (this one only has 128), and much larger object counts, so clearly this isn't the first time dynamic sharding has been invoked on this cluster. Has anyone seen this? I expect it will come up again, and can turn up debugging if that'll help. Thanks for any assistance! Josh

1 1

ceph-mon store.db disk usage increase on OSD-Host fail
by Hartwig Hauschild 12 Mar '20

12 Mar '20

Hi, I'm (still) testing upgrading from Luminous to Nautilus and ran into the following situation: The lab-setup I'm testing in has three OSD-Hosts. If one of those hosts dies the store.db in /var/lib/ceph/mon/ on all my Mon-Nodes starts to rapidly grow in size until either the OSD-host comes back up or disks are full. On another cluster that's still on Luminous I don't see any growth at all. Is that a difference in behaviour between Luminous and Nautilus or is that caused by the lab-setup only having three hosts and one lost host causing all PGs to be degraded at the same time? -- Cheers, Hardy

3 7

Single machine / multiple monitors
by Brian Topping 12 Mar '20

12 Mar '20

Hi, I’m getting conflicting reads from the documentation. It seems that by using the “cluster name”[1], multiple clusters can be run in parallel on the same hardware. In trying to set this up with `ceph-deploy`, I see the man page[2] says "if it finds the distro.init to be sysvinit (Fedora, CentOS/RHEL etc), it doesn't allow installation with custom cluster name and uses the default name ceph for the cluster”. Is it possible to run multiple clusters on the same hardware with CentOS 7 as the base OS? Thanks, Brian [1] https://docs.ceph.com/docs/nautilus/install/manual-deployment/#monitor-boot… [2] https://docs.ceph.com/docs/nautilus/man/8/ceph-deploy/?highlight=ceph-deplo…

2 2

Rados example: create namespace, user for this namespace, read and write objects with created namespace and user
by Rodrigo Severo - Fábrica 11 Mar '20

11 Mar '20

Hi, I'm trying to create a namespace in rados, create a user that has access to this created namespace and with rados command line utility read and write objects in this created namespace using the created user. I can't find an example on how to do it. Can someone point me to such example or show me how to do it? Regards, Rodrigo Severo

3 3

Setting user in rados command line utility
by Rodrigo Severo - Fábrica 11 Mar '20

11 Mar '20

Hi, How can I set the user to be used by rados command line utility? I can see no option for that in https://docs.ceph.com/docs/nautilus/man/8/rados/ Regards, Rodrigo Severo

2 2

Re: Nautilus OSD memory consumption?
by Nigel Williams 11 Mar '20

11 Mar '20

On Thu, 27 Feb 2020 at 06:27, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > If the heap stats reported by telling the OSD `heap stats` is large, telling each `heap release` may be useful. I suspect a TCMALLOC shortcoming. osd.158 tcmalloc heap stats:------------------------------------------------ MALLOC: 5722761448 ( 5457.7 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 311621760 ( 297.2 MiB) Bytes in central cache freelist MALLOC: + 26242992 ( 25.0 MiB) Bytes in transfer cache freelist MALLOC: + 62721768 ( 59.8 MiB) Bytes in thread cache freelists MALLOC: + 113340608 ( 108.1 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 6236688576 ( 5947.8 MiB) Actual memory used (physical + swap) MALLOC: + 21415870464 (20423.8 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 27652559040 (26371.5 MiB) Virtual address space used MALLOC: MALLOC: 394518 Spans in use MALLOC: 37 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory.

2 2

Accidentally removed client.admin caps - fix via mon doesn't work
by wittler＠appvisory.com 11 Mar '20

11 Mar '20

Hello Guys, Unfortunately, I''ve deleted some caps from client.admin and tried the following solution so set them back: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015474.html I’ve tried the following: # ssh’d to a mon node and changed dir to the mon directory cd /var/lib/ceph/mon/<monname> # tried to authenticate with the monitor keyring and set the client.admin caps to give back full permissions ceph -n mon.<monid> --keyring keyring auth caps client.admin mds 'allow *' osd 'allow *' mon 'allow *' When i try to modify the client.admin caps, the command just hangs at the shell until i press ctrl-c, which gets aknowledged with "Cluster connection aborted” Also i can’t see the the connection in the ceph.audit.log Do i make a mistake here or is this "workaround" not supported anymore in nautilus (14.2.6)? Regards

3 4

Re: cephfs snap mkdir strange timestamp
by Marc Roos 11 Mar '20

11 Mar '20

Sorry I am not really developer nor now details about POSIX/fs's. But if I may ask, why are you getting this time from the parent? -----Original Message----- Sent: 11 March 2020 11:32 To: Jeff Layton Cc: Marc Roos; ceph-users; ceph-devel(a)vger.kernel.org Subject: Re: [ceph-users] cephfs snap mkdir strange timestamp On Tue, Mar 10, 2020 at 01:39:29PM -0400, Jeff Layton wrote: ... > > Signed-off-by: Luis Henriques <lhenriques(a)suse.com> > > --- > > fs/ceph/inode.c | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/fs/ceph/inode.c b/fs/ceph/inode.c index > > d01710a16a4a..f4e78ade0871 100644 > > --- a/fs/ceph/inode.c > > +++ b/fs/ceph/inode.c > > @@ -82,6 +82,8 @@ struct inode *ceph_get_snapdir(struct inode *parent) > > inode->i_mode = parent->i_mode; > > inode->i_uid = parent->i_uid; > > inode->i_gid = parent->i_gid; > > + inode->i_mtime = parent->i_mtime; > > + inode->i_ctime = parent->i_ctime; > > inode->i_op = &ceph_snapdir_iops; > > inode->i_fop = &ceph_snapdir_fops; > > ci->i_snap_caps = CEPH_CAP_PIN; /* so we can open */ > > What about the atime, and the ci->i_btime ? Yeah, probably makes sense too, although the fuse client doesn't seem to touch atime (it does change btime, I missed that). I'll send v2 in a bit. Cheers, -- Luis

1 0

Bucket notification with kafka error
by 曹海旺 11 Mar '20

11 Mar '20

HI, I'm sorry to bother you again I want to use kafka to queue the notifications , I add a topic named kafka,and put the notification config xml The topic info: <ListTopicsResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/"> <ListTopicsResult> <Topics> <member> <User>sr</User> <Name>kafka</Name> <EndPoint> <EndpointAddress>kafka://192.168.3.250:9092</EndpointAddress> <EndpointArgs>kafka-ack-level=broker&push-endpoint=kafka://192.168.3.250:9092</EndpointArgs> <EndpointTopic>kafka</EndpointTopic> </EndPoint> <TopicArn>arn:aws:sns:default::kafka</TopicArn> </member> <member> <User>sr</User> <Name>kafka_kafka</Name> <EndPoint> <EndpointAddress>kafka://192.168.3.250:9092</EndpointAddress> <EndpointArgs>kafka-ack-level=broker&push-endpoint=kafka://192.168.3.250:9092</EndpointArgs> <EndpointTopic>kafka</EndpointTopic> </EndPoint> <TopicArn>arn:aws:sns:default::kafka</TopicArn> </member> <member> <User>sr</User> <Name>webno</Name> <EndPoint> <EndpointAddress>http://192.168.1.114:8080/s3/sn</EndpointAddress> <EndpointArgs>push-endpoint=http://192.168.1.114:8080/s3/sn</EndpointArgs> <EndpointTopic>webno</EndpointTopic> </EndPoint> <TopicArn>arn:aws:sns:default::webno</TopicArn> </member> </Topics> </ListTopicsResult> <ResponseMetadata> <RequestId>c4b84c5b-1e88-4f16-9863-7f68872d91a4.744394.135</RequestId> </ResponseMetadata> </ListTopicsResponse> and the put notification body : <NotificationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <TopicConfiguration> <Id>kafka</Id> <Topic>arn:aws:sns:default::kafka</Topic> </TopicConfiguration> </NotificationConfiguration> The web notification works fine ,but when I use the kafka(version 1.0 jdk 1.8) I got the debug info : 020-03-11 12:46:38.612 7fd81eeb1700 20 get_system_obj_state: s->obj_tag was set empty 2020-03-11 12:46:38.612 7fd81eeb1700 10 cache get: name=default.rgw.log++pubsub.user.sr.bucket.osstest/c4b84c5b-1e88-4f16-9863-7f68872d91a4.14175.1 : hit (requested=0x1, cached=0x17) 2020-03-11 12:46:38.612 7fd81eeb1700 20 notification: 'kafka' on topic: 'kafka' and bucket: 'osstest' (unique topic: 'kafka_kafka') apply to event of type: 's3:ObjectCreated:Put' 2020-03-11 12:46:38.612 7fd81eeb1700 1 ERROR: failed to create push endpoint: kafka://192.168.3.250:9092 due to: pubsub endpoint configuration error: unknown schema in: kafka://192.168.3.250:9092 2020-03-11 12:46:38.612 7fd81eeb1700 5 req 126 0.186s s3:put_obj WARNING: publishing notification failed, with error: -22 2020-03-11 12:46:38.612 7fd81eeb1700 2

2 1