- ceph-users - lists.ceph.io

admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
by Frank Schilder 12 Dec '24

12 Dec '24

Hi all, two days ago we upgraded our cluster from octopus to pacific. Everything went well and we see lots of improvements. Thanks for releasing the last stable version with all its fixes. I do have some questions though and this hiccup is one for starters: After the upgrade to pacific we started getting the error message "admin_socket: exception getting command descriptions: [Errno 2] No such file or directory" when using the ceph daemon command. Here is the output of a full session: [root@ceph-adm:ceph-26 ~]# ceph daemon mon.ceph-26 version | jq .release "pacific" [root@ceph-adm:ceph-26 ~]# ceph --id admin daemon mon.ceph-26 version | jq .release admin_socket: exception getting command descriptions: [Errno 2] No such file or directory [root@ceph-adm:ceph-26 ~]# ceph --id admin daemon /var/run/ceph/ceph-mon.ceph-26.asok version | jq .release "pacific" [root@ceph-adm:ceph-26 ~]# ceph daemon /var/run/ceph/ceph-mon.ceph-26.asok version | jq .release "pacific" We observe that it is impossible to use the ceph daemon command in its simple form whenever a --id argument is present. This, unfortunately, creates an unnecessary restrictions, we can't use non-admin users any more. here is why this fails: [root@ceph-adm:ceph-26 ~]# strace ceph daemon mon.ceph-26 version |& grep asok stat("/var/run/ceph/ceph-mon.ceph-26.asok", {st_mode=S_IFSOCK|0755, st_size=0, ...}) = 0 connect(3, {sa_family=AF_UNIX, sun_path="/var/run/ceph/ceph-mon.ceph-26.asok"}, 37) = 0 getpeername(3, {sa_family=AF_UNIX, sun_path="/var/run/ceph/ceph-mon.ceph-26.asok"}, [110 => 38]) = 0 connect(3, {sa_family=AF_UNIX, sun_path="/var/run/ceph/ceph-mon.ceph-26.asok"}, 37) = 0 getpeername(3, {sa_family=AF_UNIX, sun_path="/var/run/ceph/ceph-mon.ceph-26.asok"}, [110 => 38]) = 0 [root@ceph-adm:ceph-26 ~]# strace ceph --id admin daemon mon.ceph-26 version |& grep asok stat("/var/run/ceph/ceph-mon.admin.asok", 0x7fffa65e9f00) = -1 ENOENT (No such file or directory) connect(3, {sa_family=AF_UNIX, sun_path="/var/run/ceph/ceph-mon.admin.asok"}, 35) = -1 ENOENT (No such file or directory) As you can see, the daemon name "ceph-26" was replaced with the user name "admin" passed with the argument to --id. As a result the command looks for a non-existent file. Passing the full path "fixes" this. This is clearly a bug and I wonder if there is a way out, for example, by setting an explicit daemon path template in the config. I will open a tracker if a user on quincy or newer confirms that this is present in newer versions as well. I wonder if this is a fall-out of https://docs.ceph.com/en/latest/releases/pacific/#id39 Point 3: "$pid expansion in config paths like admin_socket will now properly expand to the daemon pid for commands like ceph-mds or ceph-osd. Previously only ceph-fuse/rbd-nbd expanded $pid with the actual daemon pid." Thanks for any pointers on how to work around this issue. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

MDS crashing and stuck in replay(laggy) ( "batch_ops.empty()", "p->first <= start" )
by Enrico Favero 10 Dec '24

10 Dec '24

Hi all, at the University of Zurich we run a cephfs cluster of ~12PB raw size. We currently run Pacific 16.2.15 and our clients (Ubuntu 20.04) mount cephfs using the kernel driver. The cluster was deployed in Mimic and subsequently to Nautilus (14.2.22) and then Pacific (16.2.15). Last Wednesday (4 Dec.) the kernel clients have been upgraded from 5.15.0-124 to 5.15.0-126. We run 6 active MDSs and 2 standby with static pinning of directory subtrees. Yesterday (actually it happened on Friday evening) we experienced a problem where an MDS serving rank.4 crashed and then the two standby MDS in turn crashed when trying to replay the journal. The other ranks are not affected. We would appreciate any help to debug and recover the failed filesystem. Thank you. Following some info on the crash and status of the cluster. The oldest and initial crash info is: { "assert_condition": "batch_ops.empty()", "assert_file": "/build/ceph-16.2.15/src/mds/CDentry.h", "assert_func": "virtual CDentry::~CDentry()", "assert_line": 130, "assert_msg": "/build/ceph-16.2.15/src/mds/CDentry.h: In function 'virtual CDentry::~CDentry()' thread 7f14fb70e700 time 2024-12-06T19:08:09.292770+0100\n/build/ceph-16.2.15/src/mds/CDentry.h: 130: FAILED ceph_assert(batch_ops.empty())\n", "assert_thread_name": "ms_dispatch", "backtrace": [ "/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f1507109980]", "gsignal()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f15077b4a2e]", "(ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f15077b4bb8]", "(CDentry::~CDentry()+0x4d9) [0x564e03dec519]", "(CDir::remove_dentry(CDentry*)+0x17e) [0x564e03df672e]", "(MDCache::trim_dentry(CDentry*, std::map<int, boost::intrusive_ptr<MCacheExpire>, std::less<int>, std::allocator<std::pair<int const, boost::intrusive_ptr<MCacheExpire> > > >&)+0xd4) [0x564e03cb9b54]", "(MDCache::trim_lru(unsigned long, std::map<int, boost::intrusive_ptr<MCacheExpire>, std::less<int>, std::allocator<std::pair<int const, boost::intrusive_ptr<MCacheExpire> > > >&)+0x91a) [0x564e03cbb36a]", "(MDCache::trim(unsigned long)+0xc1) [0x564e03cbbde1]", "(MDCache::upkeep_main()+0x94a) [0x564e03cf8aea]", "/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd6df) [0x7f1506c276df]", "/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f15070fe6db]", "clone()" ], "ceph_version": "16.2.15", "crash_id": "2024-12-06T18:08:09.295408Z_e6f3560d-21aa-425d-a1b2-f0b5fb5bd103", "entity_name": "mds.mds-l20-33", "os_id": "ubuntu", "os_name": "Ubuntu", "os_version": "18.04.6 LTS (Bionic Beaver)", "os_version_id": "18.04", "process_name": "ceph-mds", "stack_sig": "5e65f2db93b88e8fb398b21684d658a9ef1cec94845c09c581a0d8a80722960a", "timestamp": "2024-12-06T18:08:09.295408Z", "utsname_hostname": "mds-l20-33", "utsname_machine": "x86_64", "utsname_release": "5.4.0-146-generic", "utsname_sysname": "Linux", "utsname_version": "#163~18.04.1-Ubuntu SMP Mon Mar 20 15:02:59 UTC 2023" } and it looks similar to https://tracker.ceph.com/issues/66967 even though the "assert_line" is different. But all the following crash are different and look like: { "assert_condition": "p->first <= start", "assert_file": "/build/ceph-16.2.15/src/include/interval_set.h", "assert_func": "void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]", "assert_line": 568, "assert_msg": "/build/ceph-16.2.15/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7fdd353b9700 time 2024-12-06T19:08:24.550104+0100\n/build/ceph-16.2.15/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start)\n", "assert_thread_name": "md_log_replay", "backtrace": [ "/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fdd445bb980]", "gsignal()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fdd44c66a2e]", "(ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fdd44c66bb8]", "(interval_set<inodeno_t, std::map>::erase(inodeno_t, inodeno_t, std::function<bool (inodeno_t, inodeno_t)>)+0x296) [0x55876572c9f6]", "(interval_set<inodeno_t, std::map>::erase(inodeno_t)+0x2f) [0x55876572ca3f]", "(EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x588b) [0x558765a0868b]", "(EUpdate::replay(MDSRank*)+0x5d) [0x558765a0a0bd]", "(MDLog::_replay_thread()+0x78e) [0x55876598d64e]", "(MDLog::ReplayThread::entry()+0xd) [0x55876567f4ad]", "/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fdd445b06db]", "clone()" ], "ceph_version": "16.2.15", "crash_id": "2024-12-06T18:08:24.552401Z_35179c40-1fb8-473e-84d6-c013b27f341f", "entity_name": "mds.mds-l20-33", "os_id": "ubuntu", "os_name": "Ubuntu", "os_version": "18.04.6 LTS (Bionic Beaver)", "os_version_id": "18.04", "process_name": "ceph-mds", "stack_sig": "96ed41e917b880ad1183c7acd926e3259e37e1511eb6de7730a21d14121679b2", "timestamp": "2024-12-06T18:08:24.552401Z", "utsname_hostname": "mds-l20-33", "utsname_machine": "x86_64", "utsname_release": "5.4.0-146-generic", "utsname_sysname": "Linux", "utsname_version": "#163~18.04.1-Ubuntu SMP Mon Mar 20 15:02:59 UTC 2023" } that looks like https://tracker.ceph.com/issues/61009 even if it has not been reported for Pacific. Our current status is: root@mon-l19-31:~# ceph -s cluster: id: 6ef3c8fe-33e9-4db3-80a7-c64d8991b9f0 health: HEALTH_WARN 1054 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats 1 filesystem is degraded insufficient standby MDS daemons available 4 MDSs report slow requests 2 MDSs behind on trimming 4 pgs not deep-scrubbed in time 2 pools have too many placement groups 33 daemons have recently crashed services: mon: 3 daemons, quorum mon-l19-31,mon-l20-41,mon-l15-41 (age 2M) mgr: mon-l20-41(active, since 2M), standbys: mon-l19-31, mon-l15-41 mds: 6/6 daemons up osd: 1216 osds: 1213 up (since 11d), 1213 in (since 11d) data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 9729 pgs objects: 2.78G objects, 5.3 PiB usage: 8.4 PiB used, 3.3 PiB / 12 PiB avail pgs: 9655 active+clean 74 active+clean+scrubbing+deep root@mon-l19-31:~# ceph fs status spindlefs - 611 clients ========= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active mds-l21-34-2 Reqs: 1 /s 248k 222k 119k 5107 1 active mds-l19-33 Reqs: 1 /s 15.5M 14.5M 2049k 1836k 2 active mds-l21-33 Reqs: 0 /s 25.7M 20.5M 2765k 65.5k 3 active mds-l19-33-2 Reqs: 0 /s 20.4M 20.4M 582k 10.3k 4 replay(laggy) mds-l21-33-2 0 0 0 0 5 active mds-l21-34 Reqs: 1 /s 304k 274k 139k 9112 POOL TYPE USED AVAIL spindle_metadata metadata 592G 4321G spindle data 8384T 1092T vhp data 119T 10.2T MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) root@mon-l19-31:~# ceph health detail HEALTH_WARN 1054 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats; 1 filesystem is degraded; insufficient standby MDS daemons available; 4 MDSs report slow requests; 2 MDSs behind on trimming; 4 pgs not deep-scrubbed in time; 2 pools have too many placement groups; 33 daemons have recently crashed [WRN] BLUESTORE_NO_PER_POOL_OMAP: 1054 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats osd.13 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics ... ... ... osd.1226 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics [WRN] FS_DEGRADED: 1 filesystem is degraded fs spindlefs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more [WRN] MDS_SLOW_REQUEST: 4 MDSs report slow requests mds.mds-l21-33(mds.2): 953 slow requests are blocked > 30 secs mds.mds-l19-33(mds.1): 35 slow requests are blocked > 30 secs mds.mds-l21-34-2(mds.0): 19 slow requests are blocked > 30 secs mds.mds-l19-33-2(mds.3): 3 slow requests are blocked > 30 secs [WRN] MDS_TRIM: 2 MDSs behind on trimming mds.mds-l19-33(mds.1): Behind on trimming (449/128) max_segments: 128, num_segments: 449 mds.mds-l19-33-2(mds.3): Behind on trimming (7099/128) max_segments: 128, num_segments: 7099 [WRN] PG_NOT_DEEP_SCRUBBED: 4 pgs not deep-scrubbed in time pg 11.140a not deep-scrubbed since 2024-11-28T08:56:45.425135+0100 pg 11.694 not deep-scrubbed since 2024-11-27T22:18:56.908953+0100 pg 11.21b not deep-scrubbed since 2024-11-28T04:48:14.734599+0100 pg 11.ad8 not deep-scrubbed since 2024-11-27T19:59:32.851964+0100 [WRN] POOL_TOO_MANY_PGS: 2 pools have too many placement groups Pool spindle has 8192 placement groups, should have 8192 Pool vhp has 1024 placement groups, should have 1024 [WRN] RECENT_CRASH: 33 daemons have recently crashed mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-06T18:08:50.420719Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-06T18:08:33.409537Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-06T18:08:27.335401Z mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-06T18:08:53.156894Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-06T18:08:31.359872Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-06T18:08:29.331676Z mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-06T18:08:09.295408Z mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-06T18:08:24.552401Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-06T18:09:13.029070Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-06T18:09:10.641714Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-06T18:09:18.205911Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-06T18:09:15.176043Z mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-08T13:33:16.196251Z mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-08T13:33:18.242351Z mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-08T13:33:13.762433Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-08T15:22:20.143608Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-08T15:22:18.083712Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-08T15:22:15.248451Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:19:10.277110Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:19:06.202423Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:23:24.469758Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:23:22.372727Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:19:08.271763Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:23:26.559707Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:50:21.508069Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:50:19.417250Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:50:17.469305Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:59:42.693519Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:59:40.494913Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:59:44.763196Z and 3 more Best regards, Enrico -- Enrico Favero S3IT Services and Support for Science IT Office Y11 F 52 University of Zürich Winterthurerstrasse 190, CH-8057 Zürich (Switzerland) Tel: +41 44 635 42 22

1 0

Performance Discrepancy Between rbd bench and fio on Ceph RBD
by Ml Ml 10 Dec '24

10 Dec '24

Hi all, I have a question regarding the performance differences between rbd bench and a fio test on a Ceph RBD. During testing of my Ceph environment, I noticed that rbd bench reports significantly higher write speeds compared to fio, which is run on an XFS filesystem mounted on the same RBD. Here are the test details: fio test: ------------ fio --name=ceph_bench --rw=write --bs=4M --direct=1 --ioengine=libaio --size=4G --numjobs=1 --filename=/mnt/ceph10/backup-cluster1/testfile Results: Write speed: ~70.3 MiB/s (73.7 MB/s) IOPS: ~17 Latency: ~56.4 ms (average) rbd bench: -------------- rbd bench --io-type write --io-size 4M --io-threads 16 --io-total 12G backup-proxmox/cluster1-new --keyring /etc/ceph/ceph10/ceph.client.admin.keyring --conf /etc/ceph/ceph10/ceph.conf Results: Write speed: ~150–180 MiB/s (sustained after initial peak) IOPS: ~43–46 Threads: 16 ceph tell osd.X bench: --------------------------- osd.0: IOPS: 20.03, Throughput: 80.11 MB/s osd.1: IOPS: 23.08, Throughput: 92.33 MB/s osd.2: IOPS: 22.01, Throughput: 88.02 MB/s osd.3: IOPS: 17.65, Throughput: 70.61 MB/s osd.4: IOPS: 20.15, Throughput: 80.59 MB/s osd.5: IOPS: 19.71, Throughput: 78.82 MB/s osd.6: IOPS: 18.96, Throughput: 75.84 MB/s osd.7: IOPS: 20.13, Throughput: 80.52 MB/s osd.8: IOPS: 16.45, Throughput: 65.79 MB/s (hm) osd.9: IOPS: 31.16, Throughput: 124.65 MB/s osd.10: IOPS: 23.19, Throughput: 92.76 MB/s osd.12: IOPS: 19.10, Throughput: 76.38 MB/s osd.13: IOPS: 28.02, Throughput: 112.09 MB/s osd.14: IOPS: 19.07, Throughput: 76.27 MB/s osd.15: IOPS: 21.40, Throughput: 85.59 MB/s osd.16: IOPS: 20.24, Throughput: 80.95 MB/s I understand that rbd bench is designed to test the raw performance of the RBD layer, while fio runs at the filesystem level. However, the large discrepancy between the two results has left me curious. What could be causing this difference? Are there any specific factors related to caching, journaling, or XFS itself that could explain the significantly lower performance in the fio test? Any insights or recommendations for further troubleshooting would be greatly appreciated! Thanks in advance! Best regards, Mario

4 4

is replica pool required to store metadata for EC pool?
by Tony Liu 10 Dec '24

10 Dec '24

Hi, https://docs.ceph.com/en/reef/rados/operations/erasure-code/#erasure-coding… A replica pool is required to store metadata for EC pool? My case is RBD with OpenStack. With EC pool, the RBD image has to be created like this? rbd create --size 1G --data-pool ec_pool replicated_pool/image_name What about all other RBD operations, like snapshot, import, export, etc. Thanks! Tony

2 1

About erasure code for larger hdd
by Phong Tran Thanh 10 Dec '24

10 Dec '24

Hi community, Please help with advice on selecting an erasure coding algorithm for a 6-node cluster with 540 OSDs. What would be the appropriate values for *k* and *m*? The cluster requires a high level of HA and consistent throughput. Email: tranphong079(a)gmail.com

3 3

Single unfound object in cluster with no previous version - is there anyway to recover rather than deleting the object?
by Ivan Clayson 10 Dec '24

10 Dec '24

Hello, We have an Alma8.9 (version 4 kernel) quincy (17.2.7) CephFS cluster with spinners for our bulk data and SSDs for the metadata where we have a single unfound object in the bulk pool: [root@ceph-n30 ~]# ceph -s cluster: id: fa7cf62b-e261-49cd-b00e-383c36b79ef3 health: HEALTH_ERR 1/849660811 objects unfound (0.000%) Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 9/8468903874 objects degraded (0.000%), 1 pg degraded services: mon: 3 daemons, quorum ceph-s2,ceph-s3,ceph-s1 (age 44h) mgr: ceph-s2(active, since 45h), standbys: ceph-s3, ceph-s1 mds: 1/1 daemons up, 3 standby osd: 439 osds: 439 up (since 43h), 439 in (since 43h); 176 remapped pgs data: volumes: 1/1 healthy pools: 9 pools, 4321 pgs objects: 849.66M objects, 2.3 PiB usage: 3.0 PiB used, 1.7 PiB / 4.6 PiB avail pgs: 9/8468903874 objects degraded (0.000%) 36630744/8468903874 objects misplaced (0.433%) 1/849660811 objects unfound (0.000%) 4122 active+clean 174 active+remapped+backfill_wait 22 active+clean+scrubbing+deep 2 active+remapped+backfilling 1 active+recovery_unfound+degraded io: client: 669 MiB/s rd, 87 MiB/s wr, 302 op/s rd, 77 op/s wr recovery: 175 MiB/s, 59 objects/s [root@ceph-n30 ~]# ceph health detail | grep unfound HEALTH_ERR 1/849661114 objects unfound (0.000%); Possible data damage: 1 pg recovery_unfound; Degraded data redundancy: 9/8468906904 objects degraded (0.000%), 1 pg degraded [WRN] OBJECT_UNFOUND: 1/849661114 objects unfound (0.000%) pg 2.c90 has 1 unfound objects [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound pg 2.c90 is active+recovery_unfound+degraded, acting [259,210,390,209,43,66,322,297,25,374], 1 unfound pg 2.c90 is active+recovery_unfound+degraded, acting [259,210,390,209,43,66,322,297,25,374], 1 unfound We've tried deep-scrubbing and repairing the PG as well as rebooting the entire cluster but unfortunately this has not resolved our issue. The primary OSD (259) log reports that our 1009e1df26d.000000c9 object is missing where when we do rados commands on the object that command just hangs: [root@ceph-n30 ~]# grep 2.c90 /var/log/ceph/ceph-osd.259.log ... 2024-11-25T11:38:33.860+0000 7fd409870700 1 osd.259 pg_epoch: 512353 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512348/512349 n=211842 ec=1175/1168 lis/c=512348/472766 les/c/f=512349/472770/232522 sis=512353 pruub=11.010143280s) [259,210,390,209,43,66,322,297,NONE,374]p259(0) r=0 lpr=512353 pi=[472766,512353)/11 crt=512310'8145216 mlcod 0'0 unknown pruub 205.739364624s@ m=1 mbc={}] state<Start>: transitioning to Primary 2024-11-25T11:38:54.926+0000 7fd409870700 1 osd.259 pg_epoch: 512356 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512353/512354 n=211842 ec=1175/1168 lis/c=512353/472766 les/c/f=512354/472770/232522 sis=512356 pruub=11.945847511s) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512356 pi=[472766,512356)/10 crt=512310'8145216 mlcod 0'0 active pruub 227.741577148s@ m=1 mbc={0={(0+0)=1},1={(1+0)=1},2={(1+0)=1},3={(1+0)=1},4={(1+0)=1},5={(1+0)=1},6={(1+0)=1},7={(1+0)=1},8={(0+0)=1},9={(1+0)=1}}] start_peering_interval up [259,210,390,209,43,66,322,297,2147483647,374] -> [259,210,390,209,43,66,322,297,25,374], acting [259,210,390,209,43,66,322,297,2147483647,374] -> [259,210,390,209,43,66,322,297,25,374], acting_primary 259(0) -> 259, up_primary 259(0) -> 259, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 2024-11-25T11:38:54.926+0000 7fd409870700 1 osd.259 pg_epoch: 512356 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512353/512354 n=211842 ec=1175/1168 lis/c=512353/472766 les/c/f=512354/472770/232522 sis=512356 pruub=11.945847511s) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512356 pi=[472766,512356)/10 crt=512310'8145216 mlcod 0'0 unknown pruub 227.741577148s@ m=1 mbc={}] state<Start>: transitioning to Primary 2024-11-25T11:38:59.910+0000 7fd409870700 0 osd.259 pg_epoch: 512359 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512356/512357 n=211842 ec=1175/1168 lis/c=512356/472766 les/c/f=512357/472770/232522 sis=512356) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512356 pi=[472766,512356)/10 crt=512310'8145216 mlcod 0'0 active+recovering+degraded rops=1 m=1 mbc={0={(0+0)=1},1={(1+0)=1},2={(1+0)=1},3={(1+0)=1},4={(1+0)=1},5={(1+0)=1},6={(1+0)=1},7={(1+0)=1},8={(1+0)=1},9={(1+0)=1}} trimq=[13f6e~134]] get_remaining_shards not enough shards left to try for 2:0930c16c:::1009e1df26d.000000c9:head read result was read_result_t(r=0, errors={25(8)=-2,43(4)=-2,66(5)=-2,209(3)=-2,210(1)=-2,297(7)=-2,322(6)=-2,390(2)=-2}, noattrs, returned=(0, 8388608, [])) 2024-11-25T11:38:59.911+0000 7fd409870700 0 osd.259 pg_epoch: 512359 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512356/512357 n=211842 ec=1175/1168 lis/c=512356/472766 les/c/f=512357/472770/232522 sis=512356) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512356 pi=[472766,512356)/10 crt=512310'8145216 mlcod 0'0 active+recovering+degraded rops=1 m=1 u=1 mbc={0={(0+0)=1},1={(0+0)=1},2={(0+0)=1},3={(0+0)=1},4={(0+0)=1},5={(0+0)=1},6={(0+0)=1},7={(0+0)=1},8={(0+0)=1},9={(1+0)=1}} trimq=[13f6e~134]] on_failed_pull 2:0930c16c:::1009e1df26d.000000c9:head from shard 25(8),43(4),66(5),209(3),210(1),297(7),322(6),390(2), reps on 374(9) unfound? 1 2024-11-25T11:39:01.435+0000 7fd409870700 -1 log_channel(cluster) log [ERR] : 2.c90 has 1 objects unfound and apparently lost 2024-11-25T11:39:02.193+0000 7fd409870700 -1 log_channel(cluster) log [ERR] : 2.c90 has 1 objects unfound and apparently lost 2024-11-25T11:39:03.590+0000 7fd409870700 1 osd.259 pg_epoch: 512362 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512356/512357 n=211842 ec=1175/1168 lis/c=512356/472766 les/c/f=512357/472770/232522 sis=512362 pruub=8.352689743s) [259,210,390,209,43,66,322,297,25,NONE]p259(0) r=0 lpr=512362 pi=[472766,512362)/11 crt=512310'8145216 mlcod 0'0 active pruub 232.812744141s@ m=1 u=1 mbc={0={(0+0)=1},1={(0+0)=1},2={(0+0)=1},3={(0+0)=1},4={(0+0)=1},5={(0+0)=1},6={(0+0)=1},7={(0+0)=1},8={(0+0)=1},9={(1+0)=1}}] start_peering_interval up [259,210,390,209,43,66,322,297,25,374] -> [259,210,390,209,43,66,322,297,25,2147483647], acting [259,210,390,209,43,66,322,297,25,374] -> [259,210,390,209,43,66,322,297,25,2147483647], acting_primary 259(0) -> 259, up_primary 259(0) -> 259, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 2024-11-25T11:39:03.591+0000 7fd409870700 1 osd.259 pg_epoch: 512362 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512356/512357 n=211842 ec=1175/1168 lis/c=512356/472766 les/c/f=512357/472770/232522 sis=512362 pruub=8.352689743s) [259,210,390,209,43,66,322,297,25,NONE]p259(0) r=0 lpr=512362 pi=[472766,512362)/11 crt=512310'8145216 mlcod 0'0 unknown pruub 232.812744141s@ m=1 mbc={}] state<Start>: transitioning to Primary 2024-11-25T11:39:24.954+0000 7fd409870700 1 osd.259 pg_epoch: 512365 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512362/512363 n=211842 ec=1175/1168 lis/c=512362/472766 les/c/f=512363/472770/232522 sis=512365 pruub=11.731550217s) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512365 pi=[472766,512365)/10 crt=512310'8145216 mlcod 0'0 active pruub 257.554870605s@ m=1 mbc={0={(0+0)=1},1={(1+0)=1},2={(1+0)=1},3={(1+0)=1},4={(1+0)=1},5={(1+0)=1},6={(1+0)=1},7={(1+0)=1},8={(1+0)=1},9={(0+0)=1}}] start_peering_interval up [259,210,390,209,43,66,322,297,25,2147483647] -> [259,210,390,209,43,66,322,297,25,374], acting [259,210,390,209,43,66,322,297,25,2147483647] -> [259,210,390,209,43,66,322,297,25,374], acting_primary 259(0) -> 259, up_primary 259(0) -> 259, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 2024-11-25T11:39:24.954+0000 7fd409870700 1 osd.259 pg_epoch: 512365 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512362/512363 n=211842 ec=1175/1168 lis/c=512362/472766 les/c/f=512363/472770/232522 sis=512365 pruub=11.731550217s) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512365 pi=[472766,512365)/10 crt=512310'8145216 mlcod 0'0 unknown pruub 257.554870605s@ m=1 mbc={}] state<Start>: transitioning to Primary 2024-11-25T11:39:30.679+0000 7fd409870700 0 osd.259 pg_epoch: 512368 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512365/512366 n=211842 ec=1175/1168 lis/c=512365/472766 les/c/f=512366/472770/232522 sis=512365) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512365 pi=[472766,512365)/10 crt=512310'8145216 mlcod 0'0 active+recovering+degraded rops=1 m=1 mbc={0={(0+0)=1},1={(1+0)=1},2={(1+0)=1},3={(1+0)=1},4={(1+0)=1},5={(1+0)=1},6={(1+0)=1},7={(1+0)=1},8={(1+0)=1},9={(1+0)=1}} trimq=[13f6e~134]] get_remaining_shards not enough shards left to try for 2:0930c16c:::1009e1df26d.000000c9:head read result was read_result_t(r=0, errors={25(8)=-2,43(4)=-2,66(5)=-2,209(3)=-2,210(1)=-2,297(7)=-2,322(6)=-2,390(2)=-2}, noattrs, returned=(0, 8388608, [])) 2024-11-25T11:39:30.679+0000 7fd409870700 0 osd.259 pg_epoch: 512368 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512365/512366 n=211842 ec=1175/1168 lis/c=512365/472766 les/c/f=512366/472770/232522 sis=512365) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512365 pi=[472766,512365)/10 crt=512310'8145216 mlcod 0'0 active+recovering+degraded rops=1 m=1 u=1 mbc={0={(0+0)=1},1={(0+0)=1},2={(0+0)=1},3={(0+0)=1},4={(0+0)=1},5={(0+0)=1},6={(0+0)=1},7={(0+0)=1},8={(0+0)=1},9={(1+0)=1}} trimq=[13f6e~134]] on_failed_pull 2:0930c16c:::1009e1df26d.000000c9:head from shard 25(8),43(4),66(5),209(3),210(1),297(7),322(6),390(2), reps on 374(9) unfound? 1 2024-11-25T11:40:11.652+0000 7fd409870700 -1 log_channel(cluster) log [ERR] : 2.c90 has 1 objects unfound and apparently lost [root@ceph-n30 ~]# rados -p ec82pool stat 1009e1df26d.000000c9 ... # hanging indefinitely Similarly if we do any object store commands, then we get crashes and seg faults as well: [root@ceph-n25 ~]# ceph pg 2.c90 list_unfound { "num_missing": 1, "num_unfound": 1, "objects": [ { "oid": { "oid": "1009e1df26d.000000c9", "key": "", "snapid": -2, "hash": 914558096, "max": 0, "pool": 2, "namespace": "" }, "need": "510502'8140206", "have": "0'0", "flags": "none", "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", "locations": [ "374(9)" ] } ], "state": "NotRecovering", "available_might_have_unfound": true, "might_have_unfound": [], "more": false } [root@ceph-n11 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-374 --debug --pgid 2.c90 1009e1df26d.000000c9 dump ... -6> 2024-11-25T15:54:57.387+0000 7f8b6a0a7340 1 bluestore(/var/lib/ceph/osd/ceph-374) _upgrade_super from 4, latest 4 -5> 2024-11-25T15:54:57.387+0000 7f8b6a0a7340 1 bluestore(/var/lib/ceph/osd/ceph-374) _upgrade_super done -4> 2024-11-25T15:54:57.439+0000 7f8b5d925700 5 prioritycache tune_memory target: 4294967296 mapped: 132694016 unmapped: 618364928 heap: 751058944 old mem: 134217728 new mem: 2761652735 -3> 2024-11-25T15:54:57.439+0000 7f8b5d925700 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.0540541 -2> 2024-11-25T15:54:57.439+0000 7f8b5d925700 5 prioritycache tune_memory target: 4294967296 mapped: 132939776 unmapped: 618119168 heap: 751058944 old mem: 2761652735 new mem: 2842823159 -1> 2024-11-25T15:54:57.439+0000 7f8b5d925700 5 bluestore.MempoolThread(0x55cd3a807b40) _resize_shards cache_size: 2842823159 kv_alloc: 1241513984 kv_used: 2567024 kv_onode_alloc: 42949672 kv_onode_used: -22 meta_alloc: 1174405120 meta_used: 13360 data_alloc: 218103808 data_u sed: 0 0> 2024-11-25T15:54:57.445+0000 7f8b6a0a7340 -1 *** Caught signal (Segmentation fault) ** in thread 7f8b6a0a7340 thread_name:ceph-objectstor ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8b67941cf0] 2: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x4c) [0x55cd37e34f5c] 3: (_action_on_all_objects_in_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x13b4) [0x55cd377fdf64] 4: (action_on_all_objects_in_exact_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x64) [0x55cd377fe274] 5: main() 6: __libc_start_main() 7: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. We don't have a previous verison of this object and trying to fix-lost with the object store command seg faults: [root@ceph-n11 ~] ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-374 --pgid 2.c90 --op fix-lost --dry-run *** Caught signal (Segmentation fault) ** in thread 7f45d9890340 thread_name:ceph-objectstor ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f45d712acf0] 2: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x4c) [0x5556f3c36f5c] 3: (_action_on_all_objects_in_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x13b4) [0x5556f35fff64] 4: (action_on_all_objects_in_exact_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x64) [0x5556f3600274] 5: main() 6: __libc_start_main() 7: _start() Segmentation fault (core dumped) The recommended solution seems to be to use "mark_unfound_lost revert" but for our object, there is no previous version so I think this command will discard and delete the object. The lost object is on our live filesystem and there seems to be no easily way to find the backup version as I can't access the path and or filename associated with the object to recover it from the backup. Is there any way for us to recover this object without discarding it? Or should we just accept our losses and delete it? Kindest regards, Ivan Clayson -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N269 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

2 2

ceph multisite lifecycle not working
by Christopher Durham 10 Dec '24

10 Dec '24

I have 18.2.4 on Rocky 9 Linux.This system has been updated from octopus -> pacific -> quincy (18.2.2) -> (el8->el9 reinstall of each server, but ceph osd and mon survival) -> reef (18.2.4) over several years. It appears that I have two probably related problems with lifecycle expiration in a multsite configuration. I have two zones, one on each side of a multisite. I recently discovered (about a month after the el9 and reef 18.2.4 updates) that lifecycle expiration was (mostly) not working on the secondary zone side.I had thought initially that there may be replication issues, but while there are replication issues on individual buckets that required me to full sync individual buckets, the majority of the issues are becauselifecycle expiration is not working on the secondary side. The observation that caused me to think lifecycle is the issue is that based on a lifecycle policy for a given bucket, all objects in that bucket should be already deleted.What we are seeing is that all objects have been deleted from the bucket on the master zone, but NONE of them have been deleted on the slave side.This may vary based on the date the objects were created across multiple lifecycle runs on the master side, but objects never get deleted/expired on the slave side. I tracked this down to one of two causes, let's say for a given bucket bucket1 1. radosgw-admin lc list on the master shows that the bucket completes its lifecycle processing periodically. But on the slave side, it shows: "started": "Thu, 01 Jan 1970 ...""status": "UNINITIAL" If I run: radosgw-admin lc process --bucket bucket1 that particular bucket flushes all of its expired objects (takes awhile). But as far as I can tell at this point, it never runs lifecycle again on the slave side Now, let's say I have bucket2. 2. radosgw-admin lc list on the slave side does NOT show the bucket in the json output, yet the same command on the master side shows it! Given this, if I run radosgw-admin lc process --bucket2 causes C++ exceptions and the command crashes on the slave side (makes sense, actually) Yet in this case if I do: aws --profile bucket2_owner s3api get-bucket-lifecycle-configuration --bucket bucket2 it shows the lifecycle configuration for the bucket, regardless whether I point the awscli to the master or slave zone. In this case, if I redeploy the lifecycle with put-bucket-lifecycle-configuration to the master side, then thelifecycle status shows up in radosgw-admin lc list on the slave side (as well as on the master) as UNINITIAL, and this issue devolves to #1 above, Note that lifecycle expiration on the slave side does work for some number of buckets, but most remain in the UNINITIAL state, and others not there at all until Iredeploy the lifecycle. The slave side is a lot more active in reading and writing. So, why would the bucket not show up in lc list on the slave side, where it had before (I can't say how long ago 'before' was)?How can I get it to automatically perform lifecycle on the slave side? Would this perhaps be related to rgw_lc_max_workerrgw_lc_max_wp_workerrgw_lifecycle_work_time It appears that lifecycle processing is independent on each side, meaning that a lifecycle processing of bucket A on one side runs separately from lifecycle processing of bucket A on the other side, and as such an object may exist on one side for a time when it has been already deleted on the other side. How does rgw_lifecycle_work_time work? Does it mean that outside of the work_time window no new lifecyle processing starts, or that those in process abort/stop? Either way this may explain my observations as to too many buckets staying in UNINITIAL when those that are processing have a lot of data to delete. And why is this last one rgw_lifecycle_work_time and not rgw_lc_work_time? Anyway, any help on theses issues would be appreciated. Thanks \-Chris

2 2

Replacing Ceph Monitors for Openstack
by Adrien Georget 09 Dec '24

09 Dec '24

Hello, We are using Ceph as a storage backend for Openstack (Cinder, Nova, Glance, Manila) and we are replacing old hardware hosting Ceph monitors (MON,MGR,MDS) to new ones. I have already added the new ones in production, monitors successfully joined the quorum and new MGR/MDS are standby. For the monitors, I'm sure that the monmap is already up to date and Openstack clients are already aware of the change and it should not be a problem when I will next shut down the old monitors. The ceph.conf will be updated in all Openstack controllers to replace "mon host" with the new ones before shutting old mons down. But I have some doubts with the resilience of Openstack Manila service because the IP addresses of the monitors look hardcoded in the export location of the manila share : The manila show command returns for example : | export_locations | | | | path = 134.158.208.140:6789,134.158.208.141:6789,134.158.208.142:6789:/volumes/EC_manila/_nogroup/7a6c05d9-2fea-43b1-a6d4-06eec1e384f2 | | | share_instance_id = 7a6c05d9-2fea-43b1-a6d4-06eec1e384f2 | Has anyone already done this kind of migration in the past and can confirm my doubts? Is there any process to update shares? Cheers, Adrien

4 4

Ceph Steering Committee Election: Ceph Executive Council
by Patrick Donnelly 09 Dec '24

09 Dec '24

The Ceph Steering Committee completed an election [1] on November 18th for the next 2 year term of the Ceph Executive Council. Congratulations to Dan, Josh, and Neha on their reelection. [1] https://vote.heliosvoting.org/helios/elections/b2fa20e6-1a5d-4f49-a747-0f8f… -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Slow ops during index pool recovery causes cluster performance drop to 1%
by Szabo, Istvan (Agoda) 07 Dec '24

07 Dec '24

Hi, I'm updating from octopus to quincy and all in our cluster when index pool recovery kicks off, cluster operation drops to 1%, slow ops comes non-stop. The recovery takes 1-2 hours/nodes. What I can see the iowait on the nvme drives which belongs to the index pool is pretty high, however the throughput is less than 500MB/s, the iops is less than 5000/sec. The index pool is a 3:2 replica pool with 2048pg on 156 osd (1 nvme drive has 4 osds due to we experienced latency issue with 1 or 2 osd/nvme). If we consider let's say the nvme drive still slow with these really small load, how would that be possible to somehow ease and get rid of this cluster performance drop? If I increase replica to 4-5 would that help? It could tolerate more pg slowness maybe? FYI we have many objects in our cluster, more than 4Billions: objects: 4.06G objects, 616 TiB However I think it should still tolerate cluster recovery without penalty. What I can see in the slow osd log with default debug value is about "get_health_metrics" so far : 2024-11-02T12:38:40.762+0700 7f241bc25640 0 log_channel(cluster) log [WRN] : 6 slow requests (by type [ 'delayed' : 6 ] most affected pool [ 'hkg.rgw.buckets.index' : 6 ]) 2024-11-02T12:38:41.802+0700 7f241bc25640 -1 osd.110 626281 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.3641786447.0:2194661324 26.588 26:11aa561a:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.503457179.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e626262) 2024-11-02T12:38:41.802+0700 7f241bc25640 0 log_channel(cluster) log [WRN] : 7 slow requests (by type [ 'delayed' : 7 ] most affected pool [ 'hkg.rgw.buckets.index' : 7 ]) 2024-11-02T12:38:42.782+0700 7f241bc25640 -1 osd.110 626282 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.3641786447.0:2194661324 26.588 26:11aa561a:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.503457179.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e626262) 2024-11-02T12:38:42.782+0700 7f241bc25640 0 log_channel(cluster) log [WRN] : 7 slow requests (by type [ 'delayed' : 7 ] most affected pool [ 'hkg.rgw.buckets.index' : 7 ]) 2024-11-02T12:38:43.802+0700 7f241bc25640 -1 osd.110 626282 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.3641786447.0:2194661324 26.588 26:11aa561a:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.503457179.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e626262) 2024-11-02T12:38:43.802+0700 7f241bc25640 0 log_channel(cluster) log [WRN] : 7 slow requests (by type [ 'delayed' : 7 ] most affected pool [ 'hkg.rgw.buckets.index' : 7 ]) How we also try to make it smoother, after update and machine reboot compaction kicks off which generates 30-40 iowait on the node, we prevent with "noup" flag to put these osds into the cluster until compaction finished, however when we have 0 iowait after compaction, I unset noup so recovery can start which causes the above issue. If I wouldn't set noup it would cause even bigger issue. Thank you for help ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 15