June 2024 - ceph-users - lists.ceph.io

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

Patching Ceph cluster
by Michael Worsham 25 Apr '25

25 Apr '25

What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

6 14

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

purging already destroyed OSD leads to degraded and misplaced objects?
by tobias tempel 10 Jan '25

10 Jan '25

Dear Cephers, reorganizing one of our clusters i'm removing some hosts from it, taking "out" all OSDs on these hosts and waiting until all PGs are fine. After stopping and destroying all OSDs on one host i notice, that "purge" of such destroyed OSDs temporarily leads to degraded and misplaced objects (tried "reweight 0" as well, same picture) . Why that? I would - due to my limited experience - expect, that completely removing an already destroyed OSD could not have any effect on object placement at all. This effect yet seems not to be a real problem, as recovery works fine ... but perhaps it indicates some other issue. All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's some work to do. Perhaps you can give me a hint, where to look, for an explanation? Thank you cheers, toBias -- Tobias Tempel Deutsches Elektronen-Synchrotron - IT Notkestr. 85, 22607 Hamburg, Germany email: tobias.tempel(a)desy.de

4 7

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

rgw connection resets
by Nathan Gleason 10 Oct '24

10 Oct '24

Hello, I’ve noticed what I think may be strange behavior from some rgw transfers. It only seems to be with larger multipart mp4 files. Once a connection comes into rgw it is accepted and processed, data begins to transfer, then I see "write_data failed: Connection reset by peer” and "s3:get_obj iterate_obj() failed with -104”. The client then reconnects and the transfer seems to be ok. This doesn’t appear to happen just downloading a regular file. This is happening to both my production and test cluster. I’m not seeing errors anywhere else, the network looks good and the clusters are healthy. I’m in the process of collecting some tcpdumps but figured I would ask if anyone knows why this is occurring. We run haproxy in front of the rgw. For troubleshooting purposes we have a basic haproxy config to just one rgw. We have tested with multiple browsers. Ceph 17.2.5 - cluster built and managed with cephadm Ubuntu 20.04.6 LTS Log: 2024-01-03T17:40:23.916+0000 7f646033d700 1 ====== starting new request req=0x7f640ca95730 ===== 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s initializing for trans_id = tx000003ec1e4b4dc157efd-0065959c07-104216-default 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s rgw api priority: s3=8 s3website=7 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s host=a2-west.usacld.net 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/n athan:tshoot/testfile.mp4 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s getting op 0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.prerequest. state=0x55fa31f882e0 s->prefetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj op=21RGWGetObj_ObjStore_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying requester 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj normalizing buckets and tenants 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s->object=testfile.mp4 s->bucket=nathan/tshoot 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94200 obj=default.rgw.meta:root:nathan/tshoot state=0x55fa31f882e0 s->prefetch _data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x11, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser </ID><DisplayName>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj recalculating target 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj reading permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj WARNING: blocking librados call 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj manifest: total_size = 88811314 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: setting s->obj_tag to _0OUNxgsSsqTAs9Z6xh1SOKpJZLV_Vos 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.c om/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser</ID><DisplayNa me>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init op 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94488 obj=default.rgw.meta:users.uid:nathan$testuser state=0x55fa31f882e0 s->p refetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x13, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op mask 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj required_mask= 1 user.op_mask=7 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions begin with perm_mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_n ame=, subuser=, perm_mask=15, is_admin=0) mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for uid=anonymous 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Permissions for user not found 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for group=1 mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Found permission: 1 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask=15, is_admin=0), owner=nathan$testuser, perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask =15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op params 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj pre-executing 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj check rate limiting 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj executing 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.acl 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.content_type 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.etag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-meta-s3cmd-attrs 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s Encryption mode: 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.1 obj-ofs=0 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_1 obj-ofs=4194304 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_2 obj-ofs=8388608 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_3 obj-ofs=12582912 read_ofs=0 len=3145728 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.2 obj-ofs=15728640 read_ofs=0 len=4194304 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_1 obj-ofs=19922944 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_2 obj-ofs=24117248 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_content_length: e=0, sent=26, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::complete_header: e=0, sent=460, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::set_account: e=1 2024-01-03T17:40:24.068+0000 7f642a2d1700 4 write_data failed: Connection reset by peer 2024-01-03T17:40:24.068+0000 7f642a2d1700 0 req 4522146966294331133 0.151999518s s3:get_obj iterate_obj() failed with -104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj completing 2024-01-03T17:40:24.156+0000 7f642eada700 20 req 4522146966294331133 0.239999235s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.postrequest.nathan state=0x55fa32424760 s->prefetch_data =0 2024-01-03T17:40:24.156+0000 7f642eada700 10 req 4522146966294331133 0.239999235s cache get: name=default.rgw.log++script.postrequest.nathan : hit (negative entry) 2024-01-03T17:40:24.156+0000 7f642eada700 30 AccountingFilter::complete_request: e=1, sent=0, total=0 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj op status=-104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj http status=200 2024-01-03T17:40:24.156+0000 7f642eada700 1 ====== req done req=0x7f640ca95730 op status=-104 http_status=200 latency=0.239999235s ====== 2024-01-03T17:40:24.156+0000 7f642eada700 1 beast: 0x7f640ca95730: 10.30.0.196 - anonymous [03/Jan/2024:17:40:23.916 +0000] "GET /nathan:tshoot/testfile.mp4 HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - latency=0.239999235s 2024-01-03T17:40:24.156+0000 7f642eada700 20 failed to read header: end of stream 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT=*/* 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_ENCODING=identity;q=1, *;q=0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.7 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_HOST=a2-west.usacld.net 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_RANGE=bytes=0- 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_REFERER=https://a2-west.usacld.net/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA="Not_A Brand";v="8", "Chromium";v="120", "Brave";v="120" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_MOBILE=?0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_PLATFORM="macOS" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_DEST=video 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_MODE=no-cors 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_SITE=same-origin 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_GPC=1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_VERSION=1.1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PORT=443 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PROTO=https 2024-01-03T17:40:24.160+0000 7f6503483700 20 REMOTE_ADDR=10.30.0.196 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_METHOD=GET 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SCRIPT_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SERVER_PORT=8000 2024-01-03T17:40:24.160+0000 7f6503483700 1 ====== starting new request req=0x7f640ca95730 ===== Thanks, Nathan

2 1

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Numa pinning best practices
by Szabo, Istvan (Agoda) 15 Sep '24

15 Sep '24

Hi, Haven't really found a proper descripton in case of 2 socket how to pin osds to numa node, only this: https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployment… Tuning for All Flash Deployments - Ceph - Ceph <https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployment…> Redmine tracker.ceph.com Is there anybody have some good how to on this topic? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 5

Successfully using dm-cache
by Michael Lipp 12 Sep '24

12 Sep '24

Just in case anybody is interested: Using dm-cache works and boosts performance -- at least for my use case. The "challenge" was to get 100 (identical) Linux-VMs started on a three node hyperconverged cluster. The hardware is nothing special, each node has a Supermicro server board with a single CPU with 24 cores and 4 x 4 TB hard disks. And there's that extra 1 TB NVMe... I know that the general recommendation is to use the NVMe for WAL and metadata, but this didn't seem appropriate for my use case and I'm still not quite sure about failure scenarios with this configuration. So instead I made each drive a logical volume (managed by an OSD) and added 85 GiB NVMe to each LV as read-only cache. Each VM uses as system disk an RBD based on a snapshot from the master image. The idea was that with this configuration, all VMs should share most (actually almost all) of the data on their system disk and this data should be available from the cache. Well, it works. When booting the 100 VMs, almost all read operations are satisfied from the cache. So I get close to NVMe speed but have payed for conventional hard drives only (well, SSDs aren't that much more expensive nowadays, but the hardware is 4 years old). So, nothing sophisticated, but as I couldn't find anything about this kind of setup, it might be of interest nevertheless. - Michael

13 27

cephadm basic questions: image config, OS reimages
by Matthew Vernon 27 Aug '24

27 Aug '24

Hi, I've some experience with Ceph, but haven't used cephadm much before, and am trying to configure a pair of reef clusters with cephadm. A couple of newbie questions, if I may: * cephadm shell image I'm in an isolated environment, so pulling from a local repository. I bootstrapped OK with cephadm --image docker-registry.wikimedia.org/ceph bootstrap ... And that worked nicely, but if I want to run cephadm shell (to do any sort of admin), then I have to specify cephadm --image docker-registry.wikimedia.org/ceph shell (otherwise it just hangs failing to talk to quay.io). I found the docs, which refer to setting lots of other images, but not the one that cephadm uses: https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-en… I found an old tracker in this area: https://tracker.ceph.com/issues/47274 ...but is there a good way to arrange for cephadm to use the already-downloaded image without having to remember to specify --image each time? * OS reimages We do OS upgrades by reimaging the server (which doesn't touch the storage disks); on an old-style deployment you could then use ceph-volume to re-start the OSDs and away you went; how does one do this in a cephadm cluster? [I presume involves telling cephadm to download a new image for podman to use and suchlike] Would the process be smoother if we arranged to leave /var/lib/ceph intact between reimages? Thanks, Matthew

3 4

Public Swift bucket with Openstack Keystone integration - not working in quincy/reef
by Bartosz Bezak 31 Jul '24

31 Jul '24

Hi, Similar case as with previously fixed https://tracker.ceph.com/issues/48382 - https://github.com/ceph/ceph/pull/47308. Confirmed on Cephadm deployed Ceph 18.2.2/17.2.7 with Openstack Antelope/Yoga. I’m getting "404 NoSuchBucket" error with public buckets. Enabled with Swift/Keystone integration - everything else works fine. With rgw_swift_account_in_url = true and proper endpoints: "https://rgw.test/swift/v1/AUTH_%(project_id)s" ticking public access in horizon properly sets ACL on the bucket according to swift client: swift -v stat test-bucket URL: https://rgw.test/swift/v1/AUTH_daksjhdkajdshda/testbucket Auth Token: Account: AUTH_daksjhdkajdshda Container: testbucket Objects: 1 Bytes: 1021036 Read ACL: .r:*,.rlistings Write ACL: Sync To: Sync Key: X-Timestamp: 1710947159.41219 X-Container-Bytes-Used-Actual: 1024000 X-Storage-Policy: default-placement X-Storage-Class: STANDARD Last-Modified: Thu, 21 Mar 2024 10:30:05 GMT X-Trans-Id: tx00000092ac12312312312-1231231231-1701e5-default X-Openstack-Request-Id: tx00000092ac12312312312-1231231231-1701e5-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8 however still getting 404 NoSuchBucket error Could someone using the latest version of Ceph with Swift/Keystone integration please test public buckets? Thank you. Best regards, Bartosz Bezak

3 3

Documentation for meaning of "tag cephfs" in OSD caps
by Petr Bena 29 Jul '24

29 Jul '24

Hello In https://docs.ceph.com/en/latest/cephfs/client-auth/ we can find that ceph fs authorize cephfs_a client.foo / r /bar rw Results in client.foo key: *key* caps: [mds] allow r, allow rw path=/bar caps: [mon] allow r caps: [osd] allow rw tag cephfs data=cephfs_a What is this "tag cephfs" thing? It seems like some undocumented black magic to me, since I can't find anything that documents it. Can someone explain how it works under the hood? What does it expand to? What does it limit and how?

4 3

pg's stuck activating on osd create
by Richard Bade 18 Jul '24

18 Jul '24

Hi Everyone, I had an issue last night when I was bringing online some osds that I was rebuilding. When the osds created and came online 15pgs got stuck in activating. The first osd (osd.112) seemed to come online ok, but the second one (osd.113) triggered the issue. All the pgs in activating included osd.112 in the pg map and I resolved it by doing pg-upmap-items to map the pg back from osd.112 to where it currently was but it was painful having 10min of stuck i/o os an rbd pool with vm's running. Some details about the cluster: Pacific 16.2.15, upgraded from Nautilus fairly recently and Luminos back in the past. All osds were rebuilt on bluestore in Nautilus, as were the mons. The disks in question are Intel DC P4510 8TB nvme. I'm rebuilding them as I had previously had 4x2TB osd's per disk and now wanted to consolidate down to one osd per disk. There's around 300 osd's in the pool with 16384 pgs which means that the 2TB osds had 157pgs on them. However this means that the 8TB osds have 615pgs on them and I'm wondering if this is maybe the cause of the problem. There are no warnings about too many pgs per osd in the logs or ceph status. I have the default value of 250 for mon_max_pg_per_osd and default value of 3.0 for osd_max_pg_per_osd_hard_ratio. My plan is to reduce the number of pgs in the pool but I want to understand and prove what happened here. Is it likely I've hit pg overdose protection? If I have, how would I tell as I can't see anything in the cluster logs. Thanks, Rich

2 1

Separated multisite sync and user traffic, doable?
by Szabo, Istvan (Agoda) 17 Jul '24

17 Jul '24

Hi, Could that cause any issue if the endpoints defined in the zonegroups are not in the endpoint list behind haproxy? The question is mainly about the role of the endpoint servers in the zonegroup list. Their role is the sync only or something else also? This would be the scenario, could it work? * * I have 3 mon/mgr server and 15 OSD * RGWs on the mon/mgr would be in the zonegroup definition like this "zones": [ { "id": "61c9sdf40-fdsd-4sdd-9rty9-ed56jda41817", "name": "dc", "endpoints": [ "http://mon1:8080", "http://mon2:8080", "http://mon3:8080" ], * However for user traffic I'd use an haproxy endpoint with the 15 OSD node rgws (each osd node 1x). Ty ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Large omap in index pool even if properly sharded and not "OVER"
by Szabo, Istvan (Agoda) 16 Jul '24

16 Jul '24

Hi, I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects. Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK" This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1" This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920 2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458 Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing? Thank you in advance for your help. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 14

Re: CephFS metadata pool size
by Lars Köppel 15 Jul '24

15 Jul '24

Hello everyone, short update to this problem. The zapped OSD is rebuilt and it has now 1.9 TiB (the expected size ~50%). The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They jumped up and down a lot but the higher one has now also reached 'nearfull' status. How is this possible? What is going on? Does anyone have a solution how to fix this without zapping the OSD? Best regards, Lars [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel(a)ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai

5 19

ceph orch osd rm --zap --replace leaves cluster in odd state
by Matthew Vernon 14 Jul '24

14 Jul '24

Hi, I want to prepare a failed disk for replacement. I did: ceph orch osd rm 35 --zap --replace and it's now in the state "Done, waiting for purge", with 0 pgs, and REPLACE and ZAP set to true. It's been like this for some hours, and now my cluster is unhappy: [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.35 on host moss-be1002 not managed by cephadm (the OSD is down & out) ...and also neither the disk nor the relevant NVME LV has been zapped. I have my OSDs deployed via a spec: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe" And before issuing the ceph orch osd rm I set that to be unmanaged (ceph orch set-unmanaged osd.rrd_single_NVMe), as obviously I don't want ceph to just try and re-make a new OSD on the sad disk. I'd expected from the docs[0] that what I did would leave me with a system ready for the failed disk to be swapped (and that I could then mark osd.rrd_single_NVMe as managed again, and a new OSD built), including removing/wiping the NVME lv so it can be removed. What did I do wrong? I don't much care about the OSD id (but obviously it's neater to not just incrementally increase OSD numbers every time a disk died), but I thought that telling ceph orch not to make new OSDs then using ceph orch osd rm to zap the disk and NVME lv would have been the way to go... Thanks, Matthew [0] https://docs.ceph.com/en/reef/cephadm/services/osd/#replacing-an-osd

3 3

Help needed please ! Filesystem became read-only !
by nbarbier＠deltaonline.net 14 Jul '24

14 Jul '24

Hello, First of all, thanks for reading my message. I set up a Ceph version 18.2.2 cluster with 4 nodes, everything went fine for a while, but after copying some files, the storage showed a warning status and the following message : "HEALTH_WARN: 1 MDSs are read only mds.PVE-CZ235007SH(mds.0): MDS in read-only mode". The logs are showing : Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9999> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x1000000039c object, pool 5, errno -2 Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9998> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 mds.0.189541 unhandled write error (2) No such file or directory, force readonly... After googling for a while, I did not find a hint to understand more precisely the root cause. Any help would we greatly appreciated, or even a link to post this request elsewhere if this is not the place to. Please find below additional details if needed. Thanks a lot ! Nicolas --- # ceph osd dump [...] pool 5 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 292 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.51 [...] # ceph osd lspools 1 .mgr 4 cephfs_data 5 cephfs_metadata 18 ec-pool-001-data 19 ec-pool-001-metadata # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 633 TiB 633 TiB 61 GiB 61 GiB 0 TOTAL 633 TiB 633 TiB 61 GiB 61 GiB 0 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 119 MiB 31 357 MiB 0 200 TiB cephfs_data 4 32 71 KiB 8.38k 240 KiB 0 200 TiB cephfs_metadata 5 32 329 MiB 6.56k 987 MiB 0 200 TiB ec-pool-001-data 18 32 42 GiB 15.99k 56 GiB 0 451 TiB ec-pool-001-metadata 19 32 0 B 0 0 B 0 200 TiB # ceph status cluster: id: f16f53e1-7028-440f-bf48-f99912619c33 health: HEALTH_WARN 1 MDSs are read only services: mon: 4 daemons, quorum PVE-CZ235007SG,PVE-CZ2341016V,PVE-CZ235007SH,PVE-CZ2341016T (age 35h) mgr: PVE-CZ235007SG(active, since 2d), standbys: PVE-CZ235007SH, PVE-CZ2341016T, PVE-CZ2341016V mds: 1/1 daemons up, 3 standby osd: 48 osds: 48 up (since 2d), 48 in (since 3d) data: volumes: 1/1 healthy pools: 5 pools, 129 pgs objects: 30.97k objects, 42 GiB usage: 61 GiB used, 633 TiB / 633 TiB avail pgs: 129 active+clean

4 6

reef 18.2.3 QE validation status
by Yuri Weinstein 12 Jul '24

12 Jul '24

Details of this release are summarized here: https://tracker.ceph.com/issues/65393#note-1 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke - infra issues, still trying, Laura PTL rados - Radek, Laura approved? Travis? Nizamudeen? rgw - Casey approved? fs - Venky approved? orch - Adam King approved? krbd - Ilya approved powercycle - seems fs related, Venky, Brad PTL ceph-volume - will require https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e… Guillaume is fixing it. TIA

10 29

CephFS MDS crashing during replay with standby MDSes crashing afterwards
by Ivan Clayson 10 Jul '24

10 Jul '24

Hello, We have been experiencing a serious issue with our CephFS backup cluster running quincy (version 17.2.7) on a RHEL8-derivative Linux kernel (Alma8.9, 4.18.0-513.9.1 kernel) where our MDSes for our filesystem are constantly in a "replay" or "replay(laggy)" state and keep crashing. We have a single MDS filesystem called "ceph_backup" with 2 standby MDSes along with a 2nd unused filesystem "ceph_archive" (this holds little to no data) where we are using our "ceph_backup" filesystem to backup our data and this is the one which is currently broken. The Ceph health outputs currently are: root@pebbles-s1 14:05 [~]: ceph -s cluster: id: e3f7535e-d35f-4a5d-88f0-a1e97abcd631 health: HEALTH_WARN 1 filesystem is degraded insufficient standby MDS daemons available 1319 pgs not deep-scrubbed in time 1054 pgs not scrubbed in time services: mon: 4 daemons, quorum pebbles-s1,pebbles-s2,pebbles-s3,pebbles-s4 (age 36m) mgr: pebbles-s2(active, since 36m), standbys: pebbles-s4, pebbles-s3, pebbles-s1 mds: 2/2 daemons up osd: 1380 osds: 1380 up (since 29m), 1379 in (since 3d); 37 remapped pgs data: volumes: 1/2 healthy, 1 recovering pools: 7 pools, 2177 pgs objects: 3.55G objects, 7.0 PiB usage: 8.9 PiB used, 14 PiB / 23 PiB avail pgs: 83133528/30006841533 objects misplaced (0.277%) 2090 active+clean 47 active+clean+scrubbing+deep 29 active+remapped+backfilling 8 active+remapped+backfill_wait 2 active+clean+scrubbing 1 active+clean+snaptrim io: recovery: 1.9 GiB/s, 719 objects/s root@pebbles-s1 14:09 [~]: ceph fs status ceph_backup - 0 clients =========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) pebbles-s3 0 0 0 0 POOL TYPE USED AVAIL mds_backup_fs metadata 1255G 2780G ec82_primary_fs_data data 0 2780G ec82pool data 8442T 3044T ceph_archive - 2 clients ============ RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active pebbles-s2 Reqs: 0 /s 13.4k 7105 118 2 POOL TYPE USED AVAIL mds_archive_fs metadata 5184M 2780G ec83_primary_fs_data data 0 2780G ec83pool data 138T 2767T MDS version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) root@pebbles-s1 14:09 [~]: ceph health detail | head HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available; 1319 pgs not deep-scrubbed in time; 1054 pgs not scrubbed in time [WRN] FS_DEGRADED: 1 filesystem is degraded fs ceph_backup is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more When our cluster first ran after a reboot, Ceph ran through the 2 standby MDSes, crashing them all, until it reached the final MDS and is now stuck in this "replay(laggy)" state. Putting our MDSes into debugging mode, we can see that this MDS crashed when replaying the journal for a particular inode (this is the same for all the MDSes and they all crash on the same object): ... 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay for [521,head] had [inode 0x1005ba89481 [...539,head] /cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/ auth fragtree_t(*^2 00*^3 00000*^ 4 00001*^3 00010*^4 00011*^4 00100*^4 00101*^4 00110*^4 00111*^4 01*^3 01000*^4 01001*^3 01010*^4 01011*^3 01100*^4 01101*^4 01110*^4 01111*^4 10*^3 10000*^4 10001*^4 10010*^4 10011*^4 10100*^4 10101*^3 10110*^4 10111*^4 11*^6) v10880645 f(v0 m2024-06-22 T05:41:10.213700+0100 1281276=1281276+0) n(v12 rc2024-06-22T05:41:10.213700+0100 b1348251683896 1281277=1281276+1) old_inodes=8 (iversion lock) | dirfrag=416 dirty=1 0x55770a2bdb80] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay dir 0x1005ba89481.011011000* 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay updated dir [dir 0x1005ba89481.011011000* /cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/ [2,head] auth v=436385 cv=0/0 state=107374182 4 f(v0 m2024-06-22T05:41:10.213700+0100 2502=2502+0) n(v12 rc2024-06-22T05:41:10.213700+0100 b2120744220 2502=2502+0) hs=32+33,ss=0+0 dirty=65 | child=1 0x55770ebcda80] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay added (full) [dentry #0x1/cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fracti ons_ave_Z124.mrc.teberet7.partial [539,head] auth NULL (dversion lock) v=436384 ino=(nil) state=1610612800|bottomlru | dirty=1 0x557710444500] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay added [inode 0x1005cd4fe35 [539,head] /cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_ 005006_fractions_ave_Z124.mrc.teberet7.partial auth v436384 s=0 n(v0 1=1+0) (iversion lock) cr={99995144=0-4194304@538} 0x557710438680] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.cache.ino(0x1005cd4fe35) mark_dirty_parent 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay noting opened inode [inode 0x1005cd4fe35 [539,head] /cephfs-users/afellows/Ferdos/20210625_real_DDFHFKLMT_KriosIII_K3/cryolo/test_micrographs/FoilHole_27649821_Data_27626128_2762 6130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial auth v436384 DIRTYPARENT s=0 n(v0 1=1+0) (iversion lock) cr={99995144=0-4194304@538} | dirtyparent=1 dirty=1 0x557710438680] 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay inotable tablev 3112837 <= table 3112837 2024-06-24T13:44:55.563+0100 7f8811c40700 10 mds.0.journal EMetaBlob.replay sessionmap v 1560540883, table 1560540882 prealloc [] used 0x1005cd4fe35 2024-06-24T13:44:55.563+0100 7f8811c40700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/include/interval_set.h: I n function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7f8811c40700 time 2024-06-24T13:44:55.564315+0100 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8821e814a3] 2: /usr/lib64/ceph/libceph-common.so.2(+0x269669) [0x7f8821e81669] 3: (interval_set<inodeno_t, std::map>::erase(inodeno_t, inodeno_t, std::function<bool (inodeno_t, inodeno_t)>)+0x2e5) [0x5576f9bb2885] 4: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4377) [0x5576f9eb77b7] 5: (EUpdate::replay(MDSRank*)+0x61) [0x5576f9ebbbd1] 6: (MDLog::_replay_thread()+0x7bb) [0x5576f9e4254b] 7: (MDLog::ReplayThread::entry()+0x11) [0x5576f9af5041] 8: /lib64/libpthread.so.0(+0x81ca) [0x7f8820e6f1ca] 9: clone() I've only included a short section of the crash (this is the first trace in the log with regards to the crash with a 10/20 debug_mds option). We tried deleting the 0x1005cd4fe35 object from the object store using the "rados" command but this did not allow our MDS to successfully replay. From my understanding the journal seems okay as we didn't run out of space for example on our metadata pool and "cephfs-journal-tool journal inspect" doesn't seem to think there is any damage: root@pebbles-s1 13:58 [~]: cephfs-journal-tool --rank=ceph_backup:0 journal inspect Overall journal integrity: OK root@pebbles-s1 14:04 [~]: cephfs-journal-tool --rank=ceph_backup:0 event get --inode 1101069090357 summary Events by type: OPEN: 1 UPDATE: 3 Errors: 0 root@pebbles-s1 14:05 [~]: cephfs-journal-tool --rank=ceph_backup:0 event get --inode 1101069090357 list 2024-06-22T05:41:10.214635+0100 0x51f97d4cfe35 UPDATE: (openc) test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial 2024-06-22T05:41:11.203312+0100 0x51f97d59c848 UPDATE: (check_inode_max_size) test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial 2024-06-22T05:41:15.484871+0100 0x51f97e7344cc OPEN: () FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial 2024-06-22T05:41:15.484921+0100 0x51f97e73493b UPDATE: (rename) test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc.teberet7.partial test_micrographs/FoilHole_27649821_Data_27626128_27626130_20210628_005006_fractions_ave_Z124.mrc I was wondering whether anyone had any advice for us on how we should proceed forward? We were thinking about manually applying these events (via "event apply") where failing that we could erase this problematic event with "cephfs-journal-tool --rank=ceph_backup:0 event splice --inode 1101069090357". Is this a good idea? We would rather not rebuild the entire metadata pool if we could avoid it (once was enough for us) as this cluster has ~9 PB of data on it. Kindest regards, Ivan Clayson -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N249 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

3 16

Urgent help with degraded filesystem needed
by Dietmar Rieder 10 Jul '24

10 Jul '24

Hello cephers, we have a degraded filesystem on our ceph 18.2.2 cluster and I'd need to get it up again. We have 6 MDS daemons and (3 active, each pinned to a subtree, 3 standby) It started this night, I got the first HEALTH_WARN emails saying: HEALTH_WARN --- New --- [WARN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.default.cephmon-02.duujba(mds.1): Client apollo-10:cephfs_user failing to respond to cache pressure client_id: 1962074 === Full health status === [WARN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.default.cephmon-02.duujba(mds.1): Client apollo-10:cephfs_user failing to respond to cache pressure client_id: 1962074 then it went on with: HEALTH_WARN --- New --- [WARN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded --- Cleared --- [WARN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.default.cephmon-02.duujba(mds.1): Client apollo-10:cephfs_user failing to respond to cache pressure client_id: 1962074 === Full health status === [WARN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded Then one after another MDS was going to error state: HEALTH_WARN --- Updated --- [WARN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) daemon mds.default.cephmon-01.cepqjp on cephmon-01 is in error state daemon mds.default.cephmon-02.duujba on cephmon-02 is in error state daemon mds.default.cephmon-03.chjusj on cephmon-03 is in error state daemon mds.default.cephmon-03.xcujhz on cephmon-03 is in error state === Full health status === [WARN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) daemon mds.default.cephmon-01.cepqjp on cephmon-01 is in error state daemon mds.default.cephmon-02.duujba on cephmon-02 is in error state daemon mds.default.cephmon-03.chjusj on cephmon-03 is in error state daemon mds.default.cephmon-03.xcujhz on cephmon-03 is in error state [WARN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [WARN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more In the morning then I tried to restart the MDS in error state but the kept failing. I then reduced the number of active MDS to 1 ceph fs set cephfs max_mds 1 And set the filesystem down ceph fs set cephfs down true I tried to restart the MDS again but now I'm stuck at the following status: [root@ceph01-b ~]# ceph -s cluster: id: aae23c5c-a98b-11ee-b44d-00620b05cac4 health: HEALTH_WARN 4 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available services: mon: 3 daemons, quorum cephmon-01,cephmon-03,cephmon-02 (age 2w) mgr: cephmon-01.dsxcho(active, since 11w), standbys: cephmon-02.nssigg, cephmon-03.rgefle mds: 3/3 daemons up osd: 336 osds: 336 up (since 11w), 336 in (since 3M) data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 6401 pgs objects: 284.69M objects, 623 TiB usage: 889 TiB used, 3.1 PiB / 3.9 PiB avail pgs: 6186 active+clean 156 active+clean+scrubbing 59 active+clean+scrubbing+deep [root@ceph01-b ~]# ceph health detail HEALTH_WARN 4 failed cephadm daemon(s); 1 filesystem is degraded; insufficient standby MDS daemons available [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) daemon mds.default.cephmon-01.cepqjp on cephmon-01 is in error state daemon mds.default.cephmon-02.duujba on cephmon-02 is in unknown state daemon mds.default.cephmon-03.chjusj on cephmon-03 is in error state daemon mds.default.cephmon-03.xcujhz on cephmon-03 is in error state [WRN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more [root@ceph01-b ~]# [root@ceph01-b ~]# ceph fs status cephfs - 40 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 resolve default.cephmon-02.nyfook 12.3k 11.8k 3228 0 1 replay(laggy) default.cephmon-02.duujba 0 0 0 0 2 resolve default.cephmon-01.pvnqad 15.8k 3541 1409 0 POOL TYPE USED AVAIL ssd-rep-metadata-pool metadata 295G 63.5T sdd-rep-data-pool data 10.2T 84.6T hdd-ec-data-pool data 808T 1929T MDS version: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) The end log file of the replay(laggy) default.cephmon-02.duujba shows: [...] -11> 2024-06-19T07:12:38.980+0000 7f90fd117700 1 mds.1.journaler.pq(ro) _finish_probe_end write_pos = 8673820672 (header had 8623488918). recovered. -10> 2024-06-19T07:12:38.980+0000 7f90fd117700 4 mds.1.purge_queue operator(): open complete -9> 2024-06-19T07:12:38.980+0000 7f90fd117700 4 mds.1.purge_queue operator(): recovering write_pos -8> 2024-06-19T07:12:39.015+0000 7f9104926700 10 monclient: get_auth_request con 0x55a93ef42c00 auth_method 0 -7> 2024-06-19T07:12:39.025+0000 7f9105928700 10 monclient: get_auth_request con 0x55a93ef43400 auth_method 0 -6> 2024-06-19T07:12:39.038+0000 7f90fd117700 4 mds.1.purge_queue _recover: write_pos recovered -5> 2024-06-19T07:12:39.038+0000 7f90fd117700 1 mds.1.journaler.pq(ro) set_writeable -4> 2024-06-19T07:12:39.044+0000 7f9105127700 10 monclient: get_auth_request con 0x55a93ef43c00 auth_method 0 -3> 2024-06-19T07:12:39.113+0000 7f9104926700 10 monclient: get_auth_request con 0x55a93ed97000 auth_method 0 -2> 2024-06-19T07:12:39.123+0000 7f9105928700 10 monclient: get_auth_request con 0x55a93e903c00 auth_method 0 -1> 2024-06-19T07:12:39.236+0000 7f90fa912700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7f90fa912700 time 2024-06-19T07:12:39.235633+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start) ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f910c722e15] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f910c722fdb] 3: (interval_set<inodeno_t, std::map>::erase(inodeno_t, inodeno_t, std::function<bool (inodeno_t, inodeno_t)>)+0x2e5) [0x55a93c0de9a5] 4: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4207) [0x55a93c3e76e7] 5: (EUpdate::replay(MDSRank*)+0x61) [0x55a93c3e9f81] 6: (MDLog::_replay_thread()+0x6c9) [0x55a93c3701d9] 7: (MDLog::ReplayThread::entry()+0x11) [0x55a93c01e2d1] 8: /lib64/libpthread.so.0(+0x81ca) [0x7f910b4c81ca] 9: clone() 0> 2024-06-19T07:12:39.236+0000 7f90fa912700 -1 *** Caught signal (Aborted) ** in thread 7f90fa912700 thread_name:md_log_replay ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: /lib64/libpthread.so.0(+0x12d20) [0x7f910b4d2d20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f910c722e6f] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f910c722fdb] 6: (interval_set<inodeno_t, std::map>::erase(inodeno_t, inodeno_t, std::function<bool (inodeno_t, inodeno_t)>)+0x2e5) [0x55a93c0de9a5] 7: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4207) [0x55a93c3e76e7] 8: (EUpdate::replay(MDSRank*)+0x61) [0x55a93c3e9f81] 9: (MDLog::_replay_thread()+0x6c9) [0x55a93c3701d9] 10: (MDLog::ReplayThread::entry()+0x11) [0x55a93c01e2d1] 11: /lib64/libpthread.so.0(+0x81ca) [0x7f910b4c81ca] 12: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/ 5 rgw_datacache 1/ 5 rgw_access 1/ 5 rgw_dbstore 1/ 5 rgw_flight 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite 0/ 5 seastore 0/ 5 seastore_onode 0/ 5 seastore_odata 0/ 5 seastore_omap 0/ 5 seastore_tm 0/ 5 seastore_t 0/ 5 seastore_cleaner 0/ 5 seastore_epm 0/ 5 seastore_lba 0/ 5 seastore_fixedkv_tree 0/ 5 seastore_cache 0/ 5 seastore_journal 0/ 5 seastore_device 0/ 5 seastore_backref 0/ 5 alienstore 1/ 5 mclock 0/ 5 cyanstore 1/ 5 ceph_exporter 1/ 5 memstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7f90fa912700 / md_log_replay 7f90fb914700 / 7f90fc115700 / MR_Finisher 7f90fd117700 / PQ_Finisher 7f90fe119700 / ms_dispatch 7f910011d700 / ceph-mds 7f9102121700 / ms_dispatch 7f9103123700 / io_context_pool 7f9104125700 / admin_socket 7f9104926700 / msgr-worker-2 7f9105127700 / msgr-worker-1 7f9105928700 / msgr-worker-0 7f910d8eab00 / ceph-mds max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.default.cephmon-02.duujba.log --- end dump of recent events --- I have no idea how to resolve this and would be grateful for any help. Dietmar

7 28

cephadm discovery service certificate absent after upgrade.
by Nicolas FOURNIL 09 Jul '24

09 Jul '24

Hello, I've just fresh upgrade from Quincy to Reef and my graphs are now blank... after investigations, it seems that discovery service is not working because of no certificate : # ceph orch sd dump cert Error EINVAL: No certificate found for service discovery Maybe an upgrade issue ? Is there a way to generate or replace the certificate properly ? Regards Nicolas F.

3 8

Phantom hosts
by Tim Holloway 09 Jul '24

09 Jul '24

It's getting worse. As many may be aware, the venerable CentOS 7 OS is hitting end-of-life in a matter of days. The easiest way to upgrade my serves has been to simply create an alternate disk with the new OS, turn my provisioning system loose on it, yank the old OS system disk and jack in the new one. However, Ceph is another matter. For that part, the simplest thing to do is to destroy the Ceph node(s) on the affected box, do the OS upgrade, then re-create the nodes. But now I have even MORE strays. The OSD on my box lives on in Ceph in the dashboard host view even though the documented removal procedures were followed and the VM itself was destroyed. Further, this last node is an RGW node and I cannot remove it from the RGW configuration. It not only shows on the dashboard, it also lists as still active on the command line and as entries in the config database no matter what I do. I really need some solution to this, as it's a major chokepoint in the upgrade process Tim

3 2

Fixing BlueFS spillover (pacific 16.2.14)
by Chris Dunlop 09 Jul '24

09 Jul '24

Hi, tl;dr why are my osds still spilling? I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few): osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size. There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand. However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact: # ceph tell osd.76 compact See attached perf-dump-76 for the perf dump output: # cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs' In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device: "db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?): # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B >> SIZE << 0 B 453 GiB 14 TiB I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76. I also tried "ceph-kvstore-tool bluestore-kv ... stats": $ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start } Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it. So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage? And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device: $ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they? Cheers, Chris

4 7

Sanity check
by Adam Witwicki 08 Jul '24

08 Jul '24

Hello, Can someone please let me know what failure domain my erasure code pool is, osd or host? We thought we testing this by turning off 2 hosts, we have had one host offline recently and the cluster was still serving clients - did we get lucky? ceph osd pool get <pool> crush_rule crush_rule: ecpool ceph osd pool get <pool> erasure_code_profile erasure_code_profile: 6-2 rule ecpool { id 3 type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } ceph osd erasure-code-profile get 6-2 crush-device-class=hdd crush-failure-domain=osd crush-root=default jerasure-per-chunk-alignment=false k=6 m=2 plugin=jerasure technique=reed_sol_van w=8 octopus Regards Adam

2 1

Pacific 16.2.15 `osd noin`
by Zakhar Kirpichenko 08 Jul '24

08 Jul '24

Hi, I'm adding a few OSDs to an existing cluster, the cluster is running with `osd noout,noin`: cluster: id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 health: HEALTH_WARN noout,noin flag(s) set Specifically `noin` is documented as "prevents booting OSDs from being marked in". But freshly added OSDs were immediately marked `up` and `in`: services: ... osd: 96 osds: 96 up (since 5m), 96 in (since 6m); 338 remapped pgs flags noout,noin # ceph osd tree in | grep -E "osd.11|osd.12|osd.26" 11 hdd 9.38680 osd.11 up 1.00000 1.00000 12 hdd 9.38680 osd.12 up 1.00000 1.00000 26 hdd 9.38680 osd.26 up 1.00000 1.00000 Is this expected behavior? Do I misunderstand the purpose of the `noin` option? Best regards, Zakhar

5 7

Viability of NVMeOF/TCP for VMWare
by Drew Weaver 01 Jul '24

01 Jul '24

Howdy, I recently saw that Ceph has a gateway which allows VMWare ESXi to connect to RBD. We had another gateway like this awhile back the ISCSI gateway. The ISCSI gateway ended up being... let's say problematic. Is there any reason to believe that NVMeOF will also end up on the floor and has anyone that uses VMWare extensively evaluated it's viability? Just curious! Thanks, -Drew

5 4

How can I increase or decrease the number of osd backfilling instantly
by Jaemin Joo 01 Jul '24

01 Jul '24

Hi All, I'd like to speed up or slow down osd recovery in ceph v18.2.1. According to the page ( https://www.suse.com/ko-kr/support/kb/doc/?id=000019693 ), I understand that osd_max_backfills, osd_recovery_max_active have to be increased or decreased. but It seems not to impact the number of osd backfilling instantly. so I tried to restart all of the osds even though I didn't need to restart osds. I wonder how long it takes to show that the number of backfilling is increased or decreased If the osds is not restarted. and Is there a way to increase or decrease the number of backfilling instantly after the configuration is changed.

2 2

ATTN: DOCS
by Lander Duncan 01 Jul '24

01 Jul '24

Good morning, Here are a few issues I noticed with the documentation: 1: Page : Hardware Recommendation ( https://docs.ceph.com/en/latest/start/hardware-recommendations/) Issue: Spelling on word "overprovsioning" 2: Page : Hardware Recommendation ( https://docs.ceph.com/en/latest/start/hardware-recommendations/) Issue: Missing word in sentence "You should also each host’s percentage of the cluster’s overall capacity." Potential fix: Add the word check between "also" and "each" 3: Page: https://docs.ceph.com/en/reef/cephfs/mount-prerequisites/ Issue: Broken Link: "You can use CephFS by mounting it to your local filesystem or by using [cephfs-shell]( https://docs.ceph.com/en/reef/cephfs/cephfs-shell)" 4: Page: https://docs.ceph.com/en/reef/rados/configuration/mon-lookup-dns/#looking-u… Issue: lacking information: As you can see in the code here: https://github.com/ceph/ceph/blob/cac7dcd63481ddbc4fe26f4b8d5918510e8be457/… in lines 790-793 and through experimentation you can use the "mon_dns_srv_name" directive to pass either the "ceph-mon" and use DNS search domains, as mentioned on this page or you can pass the name followed by an underscore to manually pass the search domain. For example: ceph-mon_example.com with records placed in _ceph-mon._example.com Have a lovely day

2 4

rbd migration / cp maintaining modified/create timestamps
by Marc 30 Jun '24

30 Jun '24

How to move an image to a different pool maintaining create/modified dates etc.

1 0

cannot delete service by ceph orchestrator
by Alex from North 29 Jun '24

29 Jun '24

Hi everybody! never seen this before and google stay silent. Just found the same question in 2021 but no answer there ((( so, by ceph orch ls I see: root@ceph1:~/ceph-rollout# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 9m ago 6d count:1 ceph-exporter 10/10 9m ago 6d * crash 10/10 9m ago 6d * grafana ?:3000 1/1 9m ago 6d count:1 mgr 3/3 9m ago 6d ceph1;ceph6;ceph10;count:3 mon 3/3 9m ago 6d ceph1;ceph6;ceph10;count:3 node-exporter ?:9100 10/10 9m ago 6d * osd.osd_using_paths 108 9m ago - <unmanaged> prometheus ?:9095 3/3 9m ago 6d ceph1;ceph6;ceph10;count:3 And then ceph orch rm <service_name> root@ceph1:~/ceph-rollout# ceph orch rm osd.osd_using_paths Invalid service 'osd.osd_using_paths'. Use 'ceph orch ls' to list available services. cannot find it for some reason. In the same time root@ceph1:~/ceph-rollout# ceph orch ls --service_name osd.osd_using_paths --export service_type: osd service_id: osd_using_paths service_name: osd.osd_using_paths unmanaged: true spec: filter_logic: AND objectstore: bluestore Why does it happen? And how to delete this not needed service? Thanks in advance.

2 2

OSD service specs in mixed environment
by Torkil Svensgaard 28 Jun '24

28 Jun '24

Hi We have a bunch of HDD OSD hosts with DB/WAL on PCI NVMe, either 2 x 3.2TB or 1 x 6.4TB. We used to have 4 SSDs pr node for journals before bluestore and those have been repurposed for an SSD pool (wear level is fine). We've been using the following service specs to avoid the PCI NVMe devices for bluestore being provisioned as OSDs: --- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 size: :1000G <-- only use devices smaller than 1TB = not PCI NVMe filter_logic: AND objectstore: bluestore --- service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290966113186 data_devices: rotational: 1 db_devices: rotational: 0 size: '1000G:' <-- only use devices larger than 1TB for DB/WAL filter_logic: AND objectstore: bluestore --- We just bought a few 7.68 TB SATA SSDs to add to the SSD pool which aren't being picked up by the osd.fast spec because they are too large and they could also be picked up as DB/WAL with the current specs. As far as I can determine there is no way to achieve what I want with the existing specs, as I can't filter on PCI vs SATA, only rotational or not, I can't use size, as it only can define an in between range, not an outside range, and I can't use filter_logic OR for the sizes because I need the rotational qualifier to be AND. I can do a osd.fast2 spec with size: 7000G: and change the db_devices size for osd.slow to something like 1000G:7000G but curious to see if anyone would have a different suggestion? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

3 7

NFSv3 on Reef
by Ramon Orrù 28 Jun '24

28 Jun '24

Hello, I’m using NFSGW on a Reef release, I managed to mount cephfs volumes over NFSv4 with no problems. Now I need to export a CephFS volume using NFSv3 (I need to mount it from a client that only supports v3), and I tried everything I can imagine to make it work. I unchecked the flag created an export: but I got the error: that sounds like a wrong configuration. I created the export using a json file like: { "access_type": "RW", "clients": [ { "access_type": "RW", "addresses": [ “XX.XX.XX.XX" ], "squash": "no_root_squash" }, { "access_type": "RW", "addresses": [ “XX.XX.XX.XX" ], "squash": "no_root_squash" } ], "cluster_id": “nfsgw", "export_id": 1, "fsal": { "fs_name": “XXX", "name": "CEPH", "user_id": "nfs.nfsgw.1" }, "path": "/", "protocols": [ 3, 4 ], "pseudo": "***", "security_label": false, "squash": "no_root_squash", "transports": [ "TCP", "UDP" ] } and applied the conf with ceph nfs export apply nfsgw -i export.yaml But /etc/ganesha/ganesha.conf still contains NFS_CORE_PARAM { Enable_NLM = false; Enable_RQUOTA = false; Protocols = 4; NFS_Port = 12049; } even after pushing a custom config: # ceph nfs cluster config get nfsgw NFS_CORE_PARAM { Enable_NLM = false; Enable_RQUOTA = false; Protocols = 3, 4; NFS_Port = 12049; } and redeploying everything with ceph orch redeploy. Any help would be very appreciated, thanks in advance… Ramon -- Ramon Orrù Servizio di Calcolo Laboratori Nazionali di Frascati Istituto Nazionale di Fisica Nucleare Via E. Fermi, 54 - 00044 Frascati (RM) Italy Tel. +39 06 9403 2345

1 0

Multisite RGW with Self-signed CA & Disconnected Upgrade
by Alex Hussein-Kershaw (HE/HIM) 28 Jun '24

28 Jun '24

Hi ceph-users! I'm going through the process of migrating to use cephadm for my clusters. Previously I used ceph-ansible. I have a few questions related to this. 1. How can I configure RGW multisite with self-signed certificates? I have prototyped the migration and redeployed RGWs. Everything is running the latest version of Reef (18.2.2). My RGWs on both sites are up and I can make requests to them, but they are failing to sync, and "radosgw-admin sync status" shows a generic input/output error. Taking some network capture I can see that the TLS handshake is failing with "Unknown CA", so it looks like the RGWs don't trust my self-signed certificate, I suppose that's not a surprise. However, I can't work out how to establish the trust. I'm running in docker containers on RHEL 9, I've tried mounting in the /etc/pki directory from the machine it's running on, which does contain the self-signed CA (I can curl from the machine), but I still see errors in my multi-site sync. 2. How can I use the "ceph orch upgrade" command in a disconnected environment? After adopting my cluster, I want to upgrade it. It works great, I run something like "ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.2" and it all just works. Awesome. My problem is that I can't reach out to the quay.io registry on my production systems to do this upgrade, they are all locked down and don't have internet access. I can install the docker image directly by copying up the file, but I don't see a way to invoke the "ceph orch upgrade" command with a locally available docker image. Am I missing the obvious? Many thanks, Alex

1 0

Huge amounts of objects orphaned by lifecycle policy.
by Adam Prycki 27 Jun '24

27 Jun '24

Hello, I have a question. Do people use rgw lifecycle policies in production? I had big hopes for this technology bug in practice it seems to be very unreliable. Recently I've been testing different pool layouts and using lifecycle policy to move data between them. Once I've checked orphaned objects I've discovered that my pools were full of orphaned objects. One pool was over 1/3 orphans by volume. Orphan object belonged to data that was moved by lifecycle. Yesterday I decided to recreate one of the pools with 3TiB of data. All 3TiB was located in a single directory of some buckets. I've created a lifecycle which should move it all to STANDARD pool and run radosgw-admin lc process --bucket. After lifecycle finished executing ceph pool still contained 1TiB of data. Removing objects from rgw-orphan-list output reduced pool size to 65GiB and 17k objects. The 17k rados __shadow objects seem to belong to s3 objects which were not moved by lifecycle. I tried lifecycle from radosgw-admin but lifecycle seems to be unable to move them. s3cmd info show that they still report old storage class. Filenames don't contain special characters other than spaces. I have directories with sequentially named objects, some of them cannot be moved by lifecycle. Deleting all the objects form original 3TiB dataset also doesn't help. After running gc and orphan finding tool there are still 1,2k rados objects which should have been deleted but are not considered orphans. I've been testing on 18.2.2. Best regards Adam Prycki

2 2

Ceph Days London CFP Deadline
by Noah Lehman 27 Jun '24

27 Jun '24

Hi everyone, This is a reminder that the deadline to submit a speaking proposal for Ceph Days London is this Sunday, June 30th. Be sure to submit a proposal before then! Details here <https://ceph.io/en/community/events/2024/ceph-days-london/>. Best, Noah

1 0

ceph rgw zone create fails EINVAL
by Matthew Vernon 27 Jun '24

27 Jun '24

Hi, I'm running cephadm/reef 18.2.2. I'm trying to set up multisite. I created realm/zonegroup/master zone OK (I think!), edited the zonegroup json to include hostnames. I have this spec file for the secondary zone: rgw_zone: codfw rgw_realm_token: "SECRET" placement: label: "rgw" [I get "SECRET" by doing ceph rgw realm tokens on the master, and C&Ping the field labelled "token"] If I then try and apply this with: ceph rgw zone create -i /root/rgw_secondary.yaml It doesn't work, and I get an unhelpful backtrace: Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call return self.func(mgr, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/rgw/module.py", line 96, in wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/rgw/module.py", line 304, in _cmd_rgw_zone_create return HandleCommandResult(retval=0, stdout=f"Zones {', '.join(created_zones)} created successfully") ^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: sequence item 0: expected str instance, int found I assume I've messed up the spec file, but it looks like the one in the docs[0]. Can anyone point me in the right direction, please? [if the underlying command emits anything useful, I can't find it in the logs] Thanks, Matthew [0] https://docs.ceph.com/en/reef/mgr/rgw/#realm-credentials-token

3 7

pg deep-scrub control scheme
by David Yang 27 Jun '24

27 Jun '24

Hello everyone. I have a cluster with 8321 pgs and recently I started to get pg not deep-scrub warnings. The reason is that I reduced max_scrub to avoid the impact of scrub on IO. Here is my current scrub configuration: ~]# ceph tell osd.1 config show|grep scrub "mds_max_scrub_ops_in_progress": "5", "mon_scrub_inject_crc_mismatch": "0.000000", "mon_scrub_inject_missing_keys": "0.000000", "mon_scrub_interval": "86400", "mon_scrub_max_keys": "100", "mon_scrub_timeout": "300", "mon_warn_pg_not_deep_scrubbed_ratio": "0.800000", "mon_warn_pg_not_scrubbed_ratio": "0.500000", "osd_debug_deep_scrub_sleep": "0.000000", "osd_deep_scrub_interval": "1296000.000000", "osd_deep_scrub_keys": "1024", "osd_deep_scrub_large_omap_object_key_threshold": "200000", "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", "osd_deep_scrub_randomize_ratio": "0.080000", "osd_deep_scrub_stride": "131072", "osd_deep_scrub_update_digest_min_age": "7200", "osd_max_scrubs": "1", "osd_requested_scrub_priority": "120", "osd_scrub_auto_repair": "false", "osd_scrub_auto_repair_num_errors": "5", "osd_scrub_backoff_ratio": "0.660000", "osd_scrub_begin_hour": "0", "osd_scrub_begin_week_day": "0", "osd_scrub_chunk_max": "25", "osd_scrub_chunk_min": "5", "osd_scrub_cost": "52428800", "osd_scrub_during_recovery": "false", "osd_scrub_end_hour": "0", "osd_scrub_end_week_day": "0", "osd_scrub_extended_sleep": "0.000000", "osd_scrub_interval_randomize_ratio": "0.500000", "osd_scrub_invalid_stats": "true", "osd_scrub_load_threshold": "0.500000", "osd_scrub_max_interval": "1296000.000000", "osd_scrub_max_preemptions": "5", "osd_scrub_min_interval": "259200.000000", "osd_scrub_priority": "5", "osd_scrub_sleep": "0.000000", I am currently trying to adjust the interval of scrub. Is there a calculation formula that can be used to easily configure the scrub/deepscrub strategy? At present, there is only the adjustment of individual values, and then it is a long wait, and there may be no progress in the end.

2 2

Unable to move realm master between zonegroups -- radosgw-admin zonegroup ignoring the --rgw-zonegroup flag?
by Tim Hunter 26 Jun '24

26 Jun '24

Hi folks, We have a number of ceph clusters organized into one realm and three zonegroups. All the clusters are running ceph 17 built with cephadm. I am trying to move the metadata master from one zonegroup (us-east-1) to zone mn1 in zonegroup us-central-1, by following the steps in the documentation <https://docs.ceph.com/en/quincy/radosgw/multisite/#changing-the-metadata-ma…>, but it doesn't work as advised. I ran these commands on a system in the `mn1` zone in the us-central-1 zonegroup. sudo radosgw-admin zone modify --rgw-zone=mn1 --master sudo radosgw-admin zonegroup modify --rgw-zonegroup=us-central-1 --master sudo radosgw-admin period update --commit When I carry out those steps, the period update gives me this error: 2024-06-26T11:19:12.256+0000 7f1f5f610e40 0 Error updating periodmap, multiple master zonegroups configured 2024-06-26T11:19:12.256+0000 7f1f5f610e40 0 master zonegroup: 9ef7877b and a3abf9e1 Here we have problem (1) that the command doesn’t move the master from a3abf9e1 (us-east-1) to 9ef7877b (us-central-1). There is a fix advised by a 2016 post to ceph-users <https://www.spinics.net/lists/ceph-users/msg30873.html> -- explicitly mark the old zonegroup master as no longer the master. sudo radosgw-admin zonegroup modify --rgw-zonegroup=us-east-1 --master=false However, the json output of this command is a modified map of the us-central-1 zonegroup, showing that the 'is_master' flag has been set back to 'false' for that zonegroup. Thus we have problem (2) that I am unable to modify two different zonegroups from the same system, leaving me unable to change the realm master. `sudo radosgw-admin zonegroup get --rgw-zonegroup=us-east-1`, when run from the us-central-1 zonegroup, just shows me the us-central-1 zonegroup details. Changing the `=` to a space does not have any effect. My brief attempts at reading the code for rgw_admin.cc on github left me unenlightened. Can anyone offer any assistance here? -- Tim Hunter Senior Infrastructure Engineer telnyx.com [image: Telnyx logo]

1 0

Slow down RGW updates via orchestrator
by Boris 26 Jun '24

26 Jun '24

Hi, we've just updated our test cluster via ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.0 During the update of the RGW service, all of the daemons went down at the same time. If I would do that on our production system it would cause a small but noticeable outage. Is there a way to only update 1 daemon at a time? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 2

Ceph Leadership Team Weekly Minutes 2024-06-17
by Laura Flores 26 Jun '24

26 Jun '24

Need to update the OS Recommendations doc to represent latest supported distros - https://docs.ceph.com/en/latest/start/os-recommendations/#platforms - PR from Zac to be reviewed CLT: https://github.com/ceph/ceph/pull/58092 arm64 CI check ready to be made required (request from Rongqi Sun, the arm check maintainer) - Agreed, although need to make sure all old issues are gone first Spam emails on various ceph lists (eg: ceph-users) - Need to improve configuration around spam check (SpamAssassin scoring seems unusually low for the types of emails coming through) - Currently version 3.4.6, which is relatively recent, but the scoring seems abnormally low for the content of the emails - Evaluate raw message headers to see this - Evaluate moderating subscription requests (require approval) [Zac] CQ#5 currently being compiled -- get your announcements in before July 10. https://pad.ceph.com/p/ceph_quarterly_2024_07 Release updates - 18.2.3: Moving along, Venky addressing issues in the fs suite - 19.1.0: Rados run scheduled; needs to be reviewed by rados - Etherpad created to track extra cherry-picks: https://pad.ceph.com/p/release-cherry-pick-coordination -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

2 1

Lot of spams on the list
by Alain Péan 25 Jun '24

25 Jun '24

Hi all, I already sent an email abbout this this morning (in France) but curiously, I never received it (blocked by my own email server From University Paris-saclay ?), although hundred of spams, seemingly coming from India, with sexual subject, have arrived in my email box (even if marked as spam by Thunderbird) . It lasts since about two weeks. I think I am not the only one on the list to get them. I am very upset to see that nothing seems to be done against these spams. Spammers should be unsubscribed for example priya.yt1 at gmail), and perhaps new subscription blocked for a time ? Best Regards, Alain -- Administrateur Système/Réseau C2N Centre de Nanosciences et Nanotechnologies (UMR 9001) Boulevard Thomas Gobert (ex Avenue de La Vauve), 91120 Palaiseau Tel : 01-70-27-06-88 Bureau A255

4 5

cephadm does not recreate OSD
by Luis Domingues 25 Jun '24

25 Jun '24

Hello all. After a disk changed, we see that cephadm does not recreate the OSD. Going all back to pvs command I ended up on this issue: https://tracker.ceph.com/issues/62862 and this PR: https://github.com/ceph/ceph/pull/53500. The PR is unfortunately closed. Is this a non-bug? I tries to replicate this pvs issue on various OS. And it looks like to be the good behavior at least on PVS side, if a lv was deleted in the middle of the disk. Example with a simple vg with 5 lvs were lv 3 was deleted. ``` root@debian:~# pvs --readonly -o pv_name,vg_name,lv_name PV VG LV /dev/vdb newvg lv1 /dev/vdb newvg lv2 /dev/vdb newvg /dev/vdb newvg lv4 /dev/vdb newvg lv5 /dev/vdb newvg ``` This output can seem weird, but if we expand the output lables ``` root@debian:~# pvs --segments -o+lv_name,seg_start_pe,segtype PV VG Fmt Attr PSize PFree Start SSize LV Start Type /dev/vdb newvg lvm2 a-- <60.00g <20.00g 0 2560 lv1 0 linear /dev/vdb newvg lvm2 a-- <60.00g <20.00g 2560 2560 lv2 0 linear /dev/vdb newvg lvm2 a-- <60.00g <20.00g 5120 2560 0 free /dev/vdb newvg lvm2 a-- <60.00g <20.00g 7680 2560 lv4 0 linear /dev/vdb newvg lvm2 a-- <60.00g <20.00g 10240 2560 lv5 0 linear /dev/vdb newvg lvm2 a-- <60.00g <20.00g 12800 2559 0 free ``` Anyway, this seems to cause the duplication. Will someone have a look into this issue? Or should we look into a workaround? Thanks, Luis

1 0

Incomplete PGs. Ceph Consultant Wanted
by cellosofia1＠gmail.com 25 Jun '24

25 Jun '24

Hi community! Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked. Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from. We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details. Thanks in advance Best Regards Pablo

6 15

Incomplete PGs. Ceph Consultant Wanted
by cellosofia1＠gmail.com 25 Jun '24

25 Jun '24

Hi community! Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked. Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from. We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details. Thanks in advance Best Regards Pablo

2 1

Test after list GC
by Anthony D'Atri 25 Jun '24

25 Jun '24

Here’s a test after de-crufting held messages. Grok the fullness. — aad

1 0

Re: Lot of spams on the list
by Anthony D'Atri 24 Jun '24

24 Jun '24

I’m not sure if I have access but I can try. > On Jun 24, 2024, at 4:37 PM, Kai Stian Olstad <kaistian(a)olstad.com> wrote: > > On 24.06.2024 19:15, Anthony D'Atri wrote: >> * Subscription is now moderated >> * The three worst spammers (you know who they are) have been removed >> * I’ve deleted tens of thousands of crufty mail messages from the queue >> The list should work normally now. Working on the backlog of held messages. 99% are bogus, but I want to be careful wrt baby and bathwater. > > Will the archive[1] also be clean up? > > [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/ > > -- > Kai Stian Olstad

1 0

June User + Dev Monthly Meeting [Recording Available]
by Noah Lehman 24 Jun '24

24 Jun '24

Hi Ceph users, A recording of June's user + developer monthly meeting is now available. Thank you to everyone who participated, asked questions, and shared insights. Your feedback is crucial to the growth and health of the ceph community! Watch it here: https://youtu.be/7D9otll-kjA?feature=shared Best, Noah

1 0

why not block gmail?
by Marc 24 Jun '24

24 Jun '24

If you don't block gmail, gmail/google will never make an effort to clean up their shit. I don't think people with a gmail.com will mind, because this is free and get somewhere else a free account. tip: google does not really know what part of their infrastructure is sending email so they use spf ~all. If you process gmail.com and force the -all manually, you block mostly spam.

6 8

[rbd mirror] integrity of journal-based image mirror
by Tony Liu 24 Jun '24

24 Jun '24

Hi, Say, the source image is being updated and data is mirrored to destination continuously. Suddenly, networking of source is down and destination will be promoted and used to restore the VM. Is that going to cause any FS issue and, for example, fsck needs to be invoked to check and repair FS? Is there any integrity check during journal-based mirror to avoid "partial" update caused by networking issue? Any insight from dev or experiences from users is appreciated. Thanks! Tony

2 2

Too much gets marked as SPAM from ceph.io.
by Roland Giesler 24 Jun '24

24 Jun '24

I have noticed a disproportionate amount of ceph.io mail lands up in spam. Checking the headers, I see: Return-Path: <ceph-users-bounces(a)ceph.io> X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on box2.fast.za.net X-Spam-Flag: YES X-Spam-Level: ******* X-Spam-Status: Yes, score=7.2 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,DMARC_FAIL_REJECT,FREEMAIL_FROM,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.4 X-Spam-Report: * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.1 SPF_PASS SPF check passed * 10 DMARC_FAIL_REJECT DMARC check failed (p=reject) * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail * provider * [caduceus42[at]aol.com] * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.0 T_SCC_BODY_TEXT_LINE No description available. * 0.1 DKIM_INVALID DKIM or DK signature exists, but is not valid * -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list * manager X-Spam-Score: 7.2 Why is this? DKIM invalid?

1 0

Call Girl in Ashok Vihar Phase 3 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – (( 9999894380 )) , Call Girls in Ashok Vihar Phase 3– We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Ashok Vihar Phase 2 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – (( 9999894380 )) , Call Girls in Ashok Vihar Phase 2 – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Ashok Vihar Phase 1💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – (( 9999894380 )) , Call Girls in Ashok Vihar Phase 1 – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in DLF Phase 5 Gurgaon 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – (( 9999894380 )) , Call Girls in DLF Phase 5 Gurgaon – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in DLF Phase 4 Gurgaon 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – (( 9999894380 )) , Call Girls in DLF Phase 4 Gurgaon – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in DLF Phase 3 Gurgaon 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – (( 9999894380 )) , Call Girls in DLF Phase 3 Gurgaon – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in DLF Phase 2 Gurgaon 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – (( 9999894380 )) , Call Girls in DLF Phase 2 Gurgaon – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in DLF Phase 1 Gurgaon 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – (( 9999894380 )) , Call Girls in DLF Phase 1 Gurgaon – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Kaushambi 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – (( 9999894380 )) , Call Girls in Kaushambi – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Ghaziabad💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – (( 9999894380 )) Call Girls in Ghaziabad – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Indirapuram 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – (( 9999894380 )) , Call Girls in Indirapuram – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Vaishali 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Vaishali – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Vaishali 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Vaishali – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Wazirabad 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Wazirabad – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Dashrath Puri 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Dashrath Puri – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Vinobapuri 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Vinobapuri – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Bharat Nagar 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Bharat Nagar – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Sarvapriya Vihar 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Sarvapriya Vihar – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Kalu Sarai 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Kalu Sarai – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Saidulajab 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Saidulajab – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Devli Road 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Devli Road – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Harkesh Nagar 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Harkesh Nagar – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Pari Chowk 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Pari Chowk – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Botanical Garden 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Botanical Garden – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Naraina Vihar 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Naraina Vihar – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Nehru Vihar 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Nehru Vihar – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Saket💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Gopalpur – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Alaknanda 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Alaknanda – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Gazipur 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Gazipur – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Shahpur Jat 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Shahpur Jat – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Ramphal Chowk 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Ramphal Chowk – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Okhla Vihar 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Okhla Vihar – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Patel Chowk 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Patel Chowk – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in ISBT 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in ISBT – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Batla House 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Batla House – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Mandi House 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Mandi House – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Jama Masjid 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Jama Masjid – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Sushant Lok 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Sushant Lok – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Manglapuri 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Manglapuri – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Hargobind Enclave 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Hargobind Enclave – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Raj Nagar 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Raj Nagar – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Burari 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Burari – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0

Call Girl in Maidan Garhi 💘 (( 9999894380 )) 💘 The Most Trusted Call Girls Service
by sunny Yt 24 Jun '24

24 Jun '24

Call Us Dinesh – 💘 (( 9999894380 )) 💘, Call Girls in Maidan Garhi – We brings offer model independent genuine call girls/Women Seeking Men Delhi Justdial escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you. We offer all types of girls of your choice with space. our escorts are fully cooperative and understand your needs. all types of call girls like Housewives, College girls, Russian girls, Muslim girls, Afghani girls, Nepali girls, south Indian girls, Punjabi girls, etc. In-Call: – You Can Reach At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: – Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Note: – Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic: – Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Delhi Ncr Our Service and Rates: – One Shot – 2000/in call (time ½ hour), 5000/out call Two shot with one girl – 3500/in call (time 1 hour), 6000/out call full body to body massage with sex call girls – 3000/in call (time 1 hour) full night for one person– 7000/in call, 10000/out call (shot limit 4 shot) full night for more than 1 person – please contact Us – 9999894380 Call Girls in Delhi at a Location :– Malviya Nagar, Saket, Hauz Khas, Near by Metro Station. We are available 24*7 all days of the year Call us – 9999894380 !! Thank you for visiting.

1 0