Hi,
I am sort of newby to RGW multisite. I guess there is an important limitation about bucket index sharding if you run multisite. I would like to learn better or correct myself. And I also want to leave a bookmark here for future cephers if possible. I apologize if this is asked before however I am not able to find a good explanation from the mailing list.
AFAIK RGW needs bucket indexes as to serve bucket listing. This is optional but mostly demanded.
Now you have to set a value for the number of shards of the bucket. This is not dynamic since multisite does not support dynamic resharding yet.
AFAIK if your bucket hits the shard limit then your bucket sync is in trouble or maybe stopped. And you should plan a cluster outage in order to reshard the bucket and resync from start.
I would like to verify if this is a correct assumption?
Here is the problem I've encounter right now.
I have 2 nautilus 14.2.9 RGW multisite clusters. Since I was not fully aware of multisite/bucket limitations, I find myself with a fairly large bucket with 256 millions of objects.
Now my problem is that bucket syncing seems to be stopped or stalled. I am %100 certain of which but I failed to figure out the exact problem from "radosgw-admin bucket sync status". It is quite possible that I maybe still don't know the proper tool to figure out the root cause.
From the RGW logs I could capture this: "check_bucket_shards: resharding needed: stats.num_objects=256166901 shard max_objects=75000000". Cluster ceph.conf includes "rgw override bucket index max shards = 750".
I suppose I need to reshard this bucket to continue syncing. And the only way to reshard is that I have to stop all RGW services of both clusters, delete the secondary zone, reshard the bucket and resync the bucket from the beginning. Or for a better solution I should divide this large bucket into smaller buckets. This might have no easy way but to migrate with some kind of S3 sync tool (rather a fast one!).
Hi all,
we are looking for a proper solution of slow_ops. When the disk failed,
node is restated ... a lot of slow operations appear. Even if disk (OSD)
or node is back again most of slow_ops are still there. On the internet
we found only advice that we have to restart monitor. But this is not
right approach. Do you have some better solution? How did you treat
slow_ops in your production clusters?
We are running the latest nautilus on all clusters.
Thank you
Cheers
Michal
Hi,
When build cluster Octopus 15.2.5 initially, here is the OSD
service spec file applied.
```
service_type: osd
service_id: osd-spec
placement:
host_pattern: ceph-osd-[1-3]
data_devices:
rotational: 1
db_devices:
rotational: 0
```
After applying it, all HDDs were added and DB of each hdd is created
on SSD.
Here is the export of OSD service spec.
```
# ceph orch ls --service_name osd.osd-spec --export
service_type: osd
service_id: osd-spec
service_name: osd.osd-spec
placement:
host_pattern: ceph-osd-[1-3]
spec:
data_devices:
rotational: 1
filter_logic: AND
objectstore: bluestore
```
Why db_devices doesn't show up there?
When I replace a disk recently, when the new disk was installed and
zapped, OSD was automatically re-created, but DB was created on HDD,
not SSD. I assume this is because of that missing db_devices?
I tried to update service spec, the same result, db_devices doesn't
show up when export it.
Is this some known issue or something I am missing?
Thanks!
Tony
So, as the subject states I have an issue with buckets returning a 404 error when they are listed immediately after being created; as well the bucket fails to be deleted if you try to delete it immediately after creation.
The behaviour is intermittent.
If I leave the bucket in place for a few minutes, the bucket behaves normally. I’m thinking this is a metadata issue or something along those lines but I’m out of my depth now.
To the best of our knowledge the cluster has not changed in any way since the same tests were run in December with no errors.
We are running Ceph 14.2.16 on all parts of the cluster.
I am using the python-swift client for the connection on a CentOS7 machine.
Can replicate the results from the mons or an external client as well.
I’m willing to share my test script as well if you would like to see how I’m generating the error.
Here is a piece of the logs in case I missed something in the interpretation (log level at 20):
14:23:17.069 7faba00df700 1 ====== starting new request req=0x55fb7a138700 =====
14:23:17.069 7faba00df700 2 req 148 0.000s initializing for trans_id = tx000000000000000000094-0060245cd5-2b8949-default
14:23:17.069 7faba00df700 10 rgw api priority: s3=8 s3website=7
14:23:17.069 7faba00df700 10 host=<NameRemoved>
14:23:17.069 7faba00df700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
14:23:17.069 7faba00df700 -1 res_query() failed
14:23:17.069 7faba00df700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/404test
14:23:17.069 7faba00df700 10 ver=v1 first=404test req=
14:23:17.069 7faba00df700 10 handler=28RGWHandler_REST_Bucket_SWIFT
14:23:17.069 7faba00df700 2 req 148 0.000s getting op 2
14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket scheduling with dmclock client=3 cost=1
14:23:17.069 7faba00df700 10 op=30RGWDeleteBucket_ObjStore_SWIFT
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying requester
14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine
14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::TempURLEngine denied with reason=-13
14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SignedTokenEngine
14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket swift_user=xmcc:swift
14:23:17.069 7faba00df700 20 build_token token=0a000000786d63633a73776966748960ea4653df708a55ae2560e58acf01
14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::SignedTokenEngine granted access
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket normalizing buckets and tenants
14:23:17.069 7faba00df700 10 s->object=<NULL> s->bucket=404test
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init permissions
14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137770 obj=default.rgw.meta:root:404test state=0x55fb7a060ac0 s->prefetch_data=0
14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+root+404test : hit (negative entry)
14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0
14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17)
14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty
14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag
14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0
14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17)
14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty
14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket recalculating target
14:23:17.069 7faba00df700 10 Starting retarget
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket reading permissions
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init op
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op mask
14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket required_mask= 4 user.op_mask=7
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op permissions
14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket -- Getting permissions begin with perm_mask=50
14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) mask=50
14:23:17.069 7faba00df700 5 Searching permissions for uid=xmcc
14:23:17.069 7faba00df700 5 Found permission: 15
14:23:17.069 7faba00df700 5 Searching permissions for group=1 mask=50
14:23:17.069 7faba00df700 5 Permissions for group not found
14:23:17.069 7faba00df700 5 Searching permissions for group=2 mask=50
14:23:17.069 7faba00df700 5 Permissions for group not found
14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0), owner=xmcc, perm=2
14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op params
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket pre-executing
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket executing
14:23:17.069 7faba00df700 0 req 148 0.000s swift:delete_bucket ERROR: bucket 404test not found
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket completing
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket op status=-2002
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket http status=404
14:23:17.069 7faba00df700 1 ====== req done req=0x55fb7a138700 op status=-2002 http_status=404 latency=0s ======
--
Mike Cave
I acknowledge and respect the Lekwungen-speaking Peoples on whose traditional territories the university stands and the Songhees, Esquimalt and WSANEC peoples whose historical relationships with the land continue to this day.
Hi,
I have a few questions about krbd on kernel 4.15
1. Does it support msgr v2? (If not which kernel supports msgr v2?)
2. If krbd is using msgr v1, does it checksum (CRC) the messages that it
sends to see for example if the write is correct or not? and if it does
checksums, If there were a problem in write how does it react to that? For
example, does it raise I/O Error or retry or...?
Thanks.
Hi,
Ceph source code contains a script called vstart.sh which allows
developers to quickly test their code using a simple deployment on your
development system.
Here: https://docs.ceph.com/en/latest//dev/quick_guide/
I am really curious that how far we can go with vstart.sh script.
While my development cluster is running, I use tools like rados bench, rbd
and rbd-nbd to benchmark simple workload and test my code. Do we have
options to change the network settings in the fake cluster built from
vstart script and later benchmark it? For example , trying 1gbit ethernet
and 10gbit ethernet.
Thanks
Hi !
I respond to the list, as it may help others.
I also reorder the response.
> On Mon, Jan 18, 2021 at 2:41 PM Gilles Mocellin <
>
> gilles.mocellin(a)nuagelibre.org> wrote:
> > Hello Cephers,
> >
> > On a new cluster, I only have 2 RBD block images, and the Dashboard
> > doesn't manage to list them correctly.
> >
> > I have this message :
> > Warning
> > Displaying previously cached data for pool veeam-repos.
> >
> > Sometime it disappears, but as soon as I reload or return to the listing
> > page, it's there.
> >
> > What I've seen, is a high CPU load due to ceph-mgr on the active
> > manager.
> > And also stack-traces like this :
[...]
> > dashboard.exceptions.ViewCacheNoDataException: ViewCache: unable to
> > retrieve data
> >
> > I also see that, when I try to edit an image :
> >
> > 2021-01-18T11:13:26.383+0100 7f00199ca700 0 [dashboard ERROR
> > frontend.error]
> > (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/#/block/rbd/edit/veeam-> > repos%252Fveeam-repo2-vol1
> > <https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/#/block/rbd/edit/veeam-repos%
> > 252Fveeam-repo2-vol1>): Cannot read property 'features_name' of
> > undefined
> >
> > TypeError: Cannot read property 'features_name' of undefined
[...]
> >
> > But that's perhaps just becaus I open an Edit window on the image and it
> > does not have the datas.
> > The Edit window is empty, and I can't edit things, especially, I wan't
> > to resize the image.
> >
[...]
> > --
> > Gilles
Le jeudi 21 janvier 2021, 21:56:58 CET Ernesto Puerta a écrit :
> Hey Gilles,
>
> If I'm not wrong, that exception (ViewCacheNoDataException) happens when
> the dashboard is unable to gather all required data from Ceph within a
> defined timeout (5 secs I think, since the UI refreshes the data every ~5
> seconds).
>
> It'd be great if you could provide the steps to reproduce it and some
> insights into your environment (number of RBD pools, number of RBD images,
> snapshots, etc.).
>
> Kind Regards,
>
> Ernesto
OK,
As it is now, it always hapens, on the image listing, I have the Warning and
the list is not always up to date, if I create an image, I must wait very long
to see it.
Also, I can not edit the 2 big images I have. Perhaps the size is important,
they are 2 images of 40 TB.
If I create a 1 GB test image, I can edit and resize it.
But impossible withe the big image, the windows opens but all the fields are
empty.
Also, if it can matter, the images use a data pool (EC 3+2).
I have 2 pools, a replicated one for metadatas veeam-repos (replic x3), and a
data pool veeam-repos.data (EC 3+2).
My cluster has 6 nodes with AMD 16 cores CPU, 128 GB RAM, 10 8 TB HDD.
So 60 OSD. Soon doubling everything to 12 nodes.
Usage, as the pool and image names can tell, is to mount RBD image as a XFS
filesystem for a Veeam Backup Repository (krbd, because nbd-rbd tailed
regularly, especially during fstrim).
Hi,
What’s the difference between data sync init and bucket sync init? Data initialise the complete cluster? Bucket only bucket?
I see when initialise finished, have shards behind but doesn’t do anything with it?
What is the proper steps to bring things back to sync?
Init
Run
Restart ??
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Hi,
What’s the difference between data sync init and bucket sync init? Data initialise the complete cluster? Bucket only bucket?
I see when initialise finished, have shards behind but doesn’t do anything with it?
What is the proper steps to bring things back to sync?
Init
Run
Restart
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
HI all,
Upgrading ceph containers is failing, how do I debug it? 'cephadm pull'
seems to work on the node, but the upgrade fails.
$ ceph orch upgrade start --ceph-version 15.2.8
"message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image
thanks
Darrin
--
CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.
