- ceph-users - lists.ceph.io

how to rbd export image from group snap?
by Timo Weingärtner 02 Nov '20

02 Nov '20

Hi, we're using rbd for VM disk images and want to make consistent backups of groups of them. I know I can create a group and make consistent snapshots of all of them: # rbd --version ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) # rbd create test_foo --size 1M # rbd create test_bar --size 1M # rbd group create test # rbd group image add test test_foo # rbd group image add test test_bar # rbd group snap create test@1 But how can I export the individual image snapshots? I tried different ways of addressing them, but nothing worked: # rbd export test_foo@1 - error setting snapshot context: (2) No such file or directory # rbd export test_foo@test/1 - rbd: error opening pool 'test_foo@test': (2) No such file or directory # rbd export rbd/test_foo@test/1 - error setting snapshot context: (2) No such file or directory # rbd export test@1/test_foo - rbd: error opening pool 'test@1': (2) No such file or directory # rbd export rbd/test@1/test_foo - rbd: error opening image test: (2) No such file or directory Am I missing something? Mit freundlichen Grüßen, Timo Weingärtner Systemadministrator -- ITscope GmbH Ludwig-Erhard-Allee 20 D-76131 Karlsruhe Tel: +49 721 62737637 Fax: +49 721 66499175 https://www.itscope.com Handelsregister: AG Mannheim, HRB 232782 Sitz der Gesellschaft: Karlsruhe Geschäftsführer: Alexander Münkel, Benjamin Mund, Stefan Reger

1 0

14.2.12 breaks mon_host pointing to Round Robin DNS entry
by Wido den Hollander 02 Nov '20

02 Nov '20

Hi, I already submitted a ticket: https://tracker.ceph.com/issues/47951 Maybe other people noticed this as well. Situation: - Cluster is running IPv6 - mon_host is set to a DNS entry - DNS entry is a Round Robin with three AAAA-records root@wido-standard-benchmark:~# ceph -s unable to parse addrs in 'mon.objects.xx.xxx.net' [errno 22] error connecting to the cluster root@wido-standard-benchmark:~# The relevant part of the ceph.conf: [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx mon_host = mon.objects.xxx.xxx.xxx ms_bind_ipv6 = true This works fine with 14.2.11 and breaks under 14.2.12 Anybody else seeing this as well? Wido

5 5

Seriously degraded performance after update to Octopus
by Martin Rasmus Lundquist Hansen 02 Nov '20

02 Nov '20

Two weeks ago we updated our Ceph cluster from Nautilus (14.2.0) to Octopus (15.2.5), an update that was long overdue. We used the Ansible playbooks to perform a rolling update and except from a few minor problems with the Ansible code, the update went well. The Ansible playbooks were also used for setting up the cluster in the first place. Before updating the Ceph software we also performed a full update of CentOS and the Linux kernel (this part of the update had already been tested on one of the OSD nodes the week before and we didn't notice any problems). However, after the update we are seeing a serious decrease in performance, more than a factor of 10x in some cases. I spend a week trying to come up with an explantion or solution, but I am completely blank. Independently of Ceph I tested the network performance and the performance of the OSD disks, and I am not really seeing any problems here. The specifications of the cluster is: - 3x Monitor nodes running mgr+mon+mds (Intel(R) Xeon(R) Silver 4108 CPU @ 1.80GHz, 16 cores, 196 GB RAM) - 14x OSD nodes, each with 18 HDDs and 1 NVME (Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz, 24 cores, 384 GB RAM) - CentOS 7.8 and Kernel 5.4.51 - 100 Gbps Infiniband We are collecting various metrics using Prometheus, and on the OSD nodes we are seeing some clear differences when it comes to CPU and Memory usage. I collected some graphs here: http://mitsted.dk/ceph . After the update the system load is highly reduced, there is almost no longer any iowait for the CPU, and the free memory is no longer used for Buffers (I can confirm that the changes in these metrics are not due to the update of CentOS or the Linux kernel). All in all, now the OSD nodes are almost completely idle all the time (and so are the monitors). On the linked page I also attached two RADOS benchmarks. The first benchmark was performed when the cluster was initially configured, and the second is the same benchmark after the update to Octopus. When comparing these two, it is clear that the performance has changed dramatically. For example, in the write test the bandwidth is reduced from 320 MB/s to 21 MB/s and the number of IOPS has also dropped significantly. I temporarily tried to disable the firewall and SELinux on all nodes to see if it made any difference, but it didn’t look like it (I did not restart any services during this test, I am not sure if that could be necessary). Any suggestions for finding the root cause of this performance decrease would be greatly appreciated.

2 1

OSD down, how to reconstruct it from its main and block.db parts ?
by Wladimir Mutel 30 Oct '20

30 Oct '20

Dear all, after breaking my experimental 1-host Ceph cluster and making one its pg 'incomplete' I left it in abandoned state for some time. Now I decided to bring it back into life and found that it can not start one of its OSDs (osd.1 to name it) "ceph osd df" shows : ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 113 MiB 4.7 GiB 1.1 TiB 59.77 0.69 102 up 1 hdd 2.84549 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 2 hdd 2.84549 1.00000 2.8 TiB 2.6 TiB 2.5 TiB 57 MiB 3.8 GiB 275 GiB 90.58 1.05 176 up 3 hdd 2.84549 1.00000 2.8 TiB 2.6 TiB 2.5 TiB 57 MiB 3.9 GiB 271 GiB 90.69 1.05 185 up 4 hdd 2.84549 1.00000 2.8 TiB 2.6 TiB 2.5 TiB 63 MiB 4.2 GiB 263 GiB 90.98 1.05 184 up 5 hdd 2.84549 1.00000 2.8 TiB 2.6 TiB 2.5 TiB 52 MiB 3.8 GiB 263 GiB 90.96 1.05 178 up 6 hdd 2.53400 1.00000 2.5 TiB 2.3 TiB 2.3 TiB 173 MiB 5.2 GiB 228 GiB 91.21 1.05 178 up 7 hdd 2.53400 1.00000 2.5 TiB 2.3 TiB 2.3 TiB 147 MiB 5.2 GiB 230 GiB 91.12 1.05 168 up TOTAL 19 TiB 17 TiB 16 TiB 662 MiB 31 GiB 2.6 TiB 86.48 MIN/MAX VAR: 0.69/1.05 STDDEV: 10.90 "ceph device ls" shows : DEVICE HOST:DEV DAEMONS LIFE EXPECTANCY GIGABYTE_GP-ASACNE2100TTTDR_SN191108950380 p10s:nvme0n1 osd.1 osd.2 osd.3 osd.4 osd.5 WDC_WD30EFRX-68N32N0_WD-WCC7K1JJXVST p10s:sdd osd.1 WDC_WD30EFRX-68N32N0_WD-WCC7K1VUYPRA p10s:sda osd.6 WDC_WD30EFRX-68N32N0_WD-WCC7K2CKX8NT p10s:sdb osd.7 WDC_WD30EFRX-68N32N0_WD-WCC7K2UD8H74 p10s:sde osd.2 WDC_WD30EFRX-68N32N0_WD-WCC7K2VFTR1F p10s:sdh osd.5 WDC_WD30EFRX-68N32N0_WD-WCC7K3CYKL87 p10s:sdf osd.3 WDC_WD30EFRX-68N32N0_WD-WCC7K6FPZAJP p10s:sdc osd.0 WDC_WD30EFRX-68N32N0_WD-WCC7K7FXSCRN p10s:sdg osd.4 In my last migration, I created a bluestore volume with external block.db like this : "ceph-volume lvm prepare --bluestore --data /dev/sdd1 --block.db /dev/nvme0n1p4" And I can see this metadata by "ceph-bluestore-tool show-label --dev /dev/ceph-e53b65ba-5eb0-44f5-9160-a2328f787a0f/osd-block-8c6324a3-0364-4fad-9dcb-81a1661ee202" : { "/dev/ceph-e53b65ba-5eb0-44f5-9160-a2328f787a0f/osd-block-8c6324a3-0364-4fad-9dcb-81a1661ee202": { "osd_uuid": "8c6324a3-0364-4fad-9dcb-81a1661ee202", "size": 3000588304384, "btime": "2020-07-12T11:34:16.579735+0300", "description": "main", "bfm_blocks": "45785344", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "65536", "bfm_size": "3000588304384", "bluefs": "1", "ceph_fsid": "49cdfe90-6f6e-4afe-8558-bf14a13aadfa", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQD9ygpf+7+MABAAqtj4y1YYgxwCaAN/jgDSwg==", "ready": "ready", "require_osd_release": "14", "whoami": "1" } } and by "ceph-bluestore-tool show-label --dev /dev/nvme0n1p4" : { "/dev/nvme0n1p4": { "osd_uuid": "8c6324a3-0364-4fad-9dcb-81a1661ee202", "size": 128025886720, "btime": "2020-07-12T11:34:16.592054+0300", "description": "bluefs db" } } As you see, their osd_uuid is equal. But when I try to start it by hand : "systemctl restart ceph-osd@1" , I get this in the logs : ("journalctl -b -u ceph-osd@1") -- Logs begin at Tue 2020-10-13 19:09:49 EEST, end at Fri 2020-10-23 16:59:38 EEST. -- жов 23 16:59:36 p10s systemd[1]: Starting Ceph object storage daemon osd.1... жов 23 16:59:36 p10s systemd[1]: Started Ceph object storage daemon osd.1. жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.943+0300 7f513cebedc0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.943+0300 7f513cebedc0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.943+0300 7f513cebedc0 -1 AuthRegistry(0x560776222940) no keyring found at /var/lib/ceph/osd/ceph-1/keyring, disabling cephx жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.943+0300 7f513cebedc0 -1 AuthRegistry(0x560776222940) no keyring found at /var/lib/ceph/osd/ceph-1/keyring, disabling cephx жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.947+0300 7f513cebedc0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.947+0300 7f513cebedc0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.947+0300 7f513cebedc0 -1 AuthRegistry(0x7fff46ea5d80) no keyring found at /var/lib/ceph/osd/ceph-1/keyring, disabling cephx жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.947+0300 7f513cebedc0 -1 AuthRegistry(0x7fff46ea5d80) no keyring found at /var/lib/ceph/osd/ceph-1/keyring, disabling cephx жов 23 16:59:36 p10s ceph-osd[3987]: failed to fetch mon config (--no-mon-config to skip) жов 23 16:59:36 p10s systemd[1]: ceph-osd(a)1.service: Main process exited, code=exited, status=1/FAILURE жов 23 16:59:36 p10s systemd[1]: ceph-osd(a)1.service: Failed with result 'exit-code'. And so my question is, how to make this OSD known again to Ceph cluster without recreating it anew with ceph-volume ? I see that every folder under "/var/lib/ceph/osd/" is a tmpfs mount point filled with appropriate files and symlinks, except of "/var/lib/ceph/osd/ceph-1", which is just an empty folder not mounted anywhere. I tried to run "ceph-bluestore-tool prime-osd-dir --dev /dev/ceph-e53b65ba-5eb0-44f5-9160-a2328f787a0f/osd-block-8c6324a3-0364-4fad-9dcb-81a1661ee202 --path /var/lib/ceph/osd/ceph-1" it created some files under /var/lib/ceph/osd/ceph-1 but without tmpfs mount, and these files belonged to root. I changed owner of these files into ceph.ceph , I created appropriate symlinks for block and block.db but ceph-osd@1 did not want to start either. Only "unable to find keyring" messages disappeared. Please give any help on where to move next. Thanks in advance for your help.

2 4

MDS_CLIENT_LATE_RELEASE: 3 clients failing to respond to capability release
by Frank Schilder 30 Oct '20

30 Oct '20

Dear cephers, I have a somewhat strange situation. I have the health warning: # ceph health detail HEALTH_WARN 3 clients failing to respond to capability release MDS_CLIENT_LATE_RELEASE 3 clients failing to respond to capability release mdsceph-12(mds.0): Client sn106.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 30716617 mdsceph-12(mds.0): Client sn269.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 30717358 mdsceph-12(mds.0): Client sn009.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 30749150 However, these clients are not busy right now. Also, they hold almost nothing; see snippets from "session ls" below. It is possible that a very IO intensive application was running on these nodes and these release requests got stuck. How do I resolve this issue? Can I just evict the client? Version is mimic 13.2.8. Note that we execute a drop cache command after a job finishes on these clients. Its possible that the clients dropped the caps already before the MDS request was handled/received. Best regards, Frank { "id": 30717358, "num_leases": 0, "num_caps": 44, "state": "open", "request_load_avg": 0, "uptime": 6632206.332307, "replay_requests": 0, "completed_requests": 0, "reconnecting": false, "inst": "client.30717358 192.168.57.140:0/3212676185", "client_metadata": { "features": "00000000000000ff", "entity_id": "con-fs2-hpc", "hostname": "sn269.hpc.ait.dtu.dk", "kernel_version": "3.10.0-957.12.2.el7.x86_64", "root": "/hpc/home" } }, -- { "id": 30716617, "num_leases": 0, "num_caps": 48, "state": "open", "request_load_avg": 1, "uptime": 6632206.336307, "replay_requests": 0, "completed_requests": 1, "reconnecting": false, "inst": "client.30716617 192.168.56.233:0/2770977433", "client_metadata": { "features": "00000000000000ff", "entity_id": "con-fs2-hpc", "hostname": "sn106.hpc.ait.dtu.dk", "kernel_version": "3.10.0-957.12.2.el7.x86_64", "root": "/hpc/home" } }, -- { "id": 30749150, "num_leases": 0, "num_caps": 44, "state": "open", "request_load_avg": 0, "uptime": 6632206.338307, "replay_requests": 0, "completed_requests": 0, "reconnecting": false, "inst": "client.30749150 192.168.56.136:0/2578719015", "client_metadata": { "features": "00000000000000ff", "entity_id": "con-fs2-hpc", "hostname": "sn009.hpc.ait.dtu.dk", "kernel_version": "3.10.0-957.12.2.el7.x86_64", "root": "/hpc/home" } }, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 3

Very high read IO during backfilling
by Kamil Szczygieł 30 Oct '20

30 Oct '20

Hi, We're running Octopus and we've 3 control plane nodes (12 core, 64 GB memory each) that are running mon, mds and mgr and also 4 data nodes (12 core, 256 GB memory, 13x10TB HDDs each). We've increased number of PGs inside our pool, which resulted in all OSDs going crazy and reading the average of 900 M/s constantly (based on iotop). This has resulted in slow ops and very low recovery speed. Any tips on how to handle this kind of situation? We've osd_recovery_sleep_hdd set to 0.2, osd_recovery_max_active set to 5 and osd_max_backfills set to 4. Some OSDs are reporting slow ops constantly and iowait on machines is at 70-80% constantly.

3 2

RBD low iops with 4k object size
by w1kl4s 30 Oct '20

30 Oct '20

Hello! I'm currently in process of playing around with ceph, and before migrating my servers to use it i wanted to benchmark it to have some idea what expected performance it might achieve. I am using 2 400GB Intel S3700 ssds with crush rule created as follows: ➜ ~ ceph osd crush rule create-replicated ssd_mirror default osd ssd And later creating pool ssd_test setting both size and min_size to 2. I proceeded to benchmark it with rbd bench, and using default image object size of 4M the result is about what i have been expecting (10k iops per OSD). However, when creating image with 4K block size, IOPS go down the drain, despite theoretically being better suited for it (4k writes should be amplified to 4M ones in case of 4M object size, reducing iops, while 4k object size should be the one without write amplification). Here is what i used to create the images and benchmark them: ➜ ~ rbd create -p ssd_test --size 16G --object-size 4K big_4k ➜ ~ rbd create -p ssd_test --size 16G --object-size 4M big_4m ➜ ~ rbd bench --io-type write --io-size 4K --io-pattern rand --io-total 256M ssd_test/big_4m bench type write io_size 4096 io_threads 16 bytes 268435456 pattern random SEC OPS OPS/SEC BYTES/SEC 1 8624 8640.00 35389458.37 2 12928 6472.00 26509325.75 3 22672 7562.67 30976698.73 4 37344 9340.00 38256659.85 5 52000 10403.21 42611529.30 elapsed: 6 ops: 65536 ops/sec: 10886.38 bytes/sec: 44590630.43 ➜ ~ rbd bench --io-type write --io-size 4K --io-pattern rand --io-total 256M ssd_test/big_4k bench type write io_size 4096 io_threads 16 bytes 268435456 pattern random SEC OPS OPS/SEC BYTES/SEC 1 6736 6645.67 27220675.51 2 7312 3664.00 15007751.77 3 7920 2645.33 10835290.95 4 8560 2135.46 8746841.19 5 9168 1828.03 7487596.24 6 9760 605.77 2481232.06 7 10368 610.22 2499477.33 8 10960 605.58 2480447.50 9 11536 596.15 2441847.42 10 12160 599.84 2456944.34 11 12816 610.22 2499477.33 12 13488 624.00 2555905.32 13 14144 636.29 2606249.15 14 14816 654.43 2680544.09 15 15472 661.87 2711022.99 16 16128 664.53 2721901.89 17 16800 662.40 2713191.81 18 17472 668.27 2737248.00 19 18080 650.72 2665341.09 20 18544 614.40 2516583.70 21 19056 584.20 2392875.94 22 19376 513.97 2105207.79 23 19584 416.73 1706936.95 24 19840 343.21 1405804.16 25 19984 287.77 1178705.65 26 20128 214.06 876780.01 27 20368 199.04 815255.64 28 20896 265.80 1088726.66 29 21488 339.65 1391222.48 30 22080 419.87 1719795.76 31 22688 512.82 2100513.91 32 23280 582.40 2385511.64 33 23888 598.88 2453010.08 34 24480 600.32 2458916.20 35 25104 603.83 2473304.79 36 25744 611.69 2505480.88 37 26432 629.39 2577994.94 38 27072 636.80 2608334.15 39 27760 653.39 2676272.30 40 28464 673.08 2756924.50 41 29104 673.08 2756924.50 42 29776 668.27 2737216.44 43 30400 665.60 2726299.01 44 31088 667.74 2735051.18 45 31744 654.43 2680544.09 46 32400 657.10 2691471.89 47 33056 657.05 2691283.44 48 33680 655.48 2684829.53 49 34304 642.69 2632442.61 50 35024 656.53 2689128.69 51 35616 645.27 2643006.18 52 36240 635.78 2604167.48 53 36880 641.03 2625642.38 54 37568 652.28 2671732.80 55 38192 634.62 2599385.96 56 38832 642.69 2632442.61 57 39472 647.44 2651898.81 58 40144 651.24 2667468.26 59 40816 648.56 2656512.56 60 41424 647.96 2654025.43 61 42112 653.39 2676272.30 62 42752 655.48 2684829.53 63 43392 649.08 2658636.07 64 44064 650.12 2662893.29 65 44704 653.91 2678406.49 66 45376 654.37 2680302.91 67 46048 658.67 2697926.26 68 46704 663.46 2717539.87 69 47344 654.95 2682685.09 70 48000 659.73 2702246.39 71 48672 660.26 2704411.65 72 49344 659.73 2702246.39 73 50032 665.07 2724119.71 74 50688 672.03 2752618.79 75 51344 667.20 2732847.38 76 52000 661.89 2711116.75 77 52592 650.12 2662893.29 78 53248 642.69 2632442.61 79 53920 645.37 2643426.29 80 54576 649.52 2660425.81 81 55232 649.00 2658288.92 82 55872 655.48 2684829.52 83 56544 661.32 2708752.60 84 57184 653.32 2676010.99 85 57856 656.00 2686977.39 86 58512 657.05 2691283.44 87 59216 668.27 2737216.44 88 59904 670.39 2745923.21 89 60592 679.97 2785150.68 90 61264 680.51 2787375.24 91 61936 684.25 2802700.09 92 62608 680.58 2787648.31 93 63248 667.73 2735030.17 94 63856 654.90 2682454.03 95 64496 645.88 2645539.34 96 65184 648.05 2654392.43 elapsed: 105 ops: 65536 ops/sec: 621.88 bytes/sec: 2547213.95 What could be causing such decrease in random write performance in case of 4k object size image? Best regards, w1kl4s

1 0

Corrupted RBD image
by Ing. Luis Felipe Domínguez Vega 30 Oct '20

30 Oct '20

Hi: I tried get info from a RBD image but: ------------------------------------------------------------------------- root@fond-beagle:/# rbd list --pool cinder-ceph | grep volume-dfcca6c8-cb96-4b79-bc85-b200a061dcda > volume-dfcca6c8-cb96-4b79-bc85-b200a061dcda root@fond-beagle:/# rbd info --pool cinder-ceph volume-dfcca6c8-cb96-4b79-bc85-b200a061dcda > rbd: error opening image volume-dfcca6c8-cb96-4b79-bc85-b200a061dcda: > (2) No such file or directory ---------------------------------------------------------------------- THis is that the metadata show the image but the content was removed?

3 3

Mon crashes when adding 4th OSD
by Lalit Maganti 30 Oct '20

30 Oct '20

(sending this email again as the first time was blocked because my attached log file was too big) Hi all, *Context*: I'm running Ceph Octopus 15.2.5 (the latest as of this email) using Rook on a toy Kubernetes cluster of two nodes. I've got a single Ceph mon node running perfectly with 3 OSDs . There are two pools running which were created as part of a CephFS install. *Problem*: when I try to add my 4th OSD, the Ceph mon starts crashing on the OSDMonitor::build_incremental function. I've checked on the mailing lists and just in general and the last instance of this issue seems to have been 7 years ago so I'm probably not hitting the same thing! *Question*: I was wondering if anyone had ideas on what I might be doing wrong? I'm very new to Ceph so my suspicion is that it's something to do with my configuration but given I'm literally just adding an OSD and everything is fine otherwise, I'm not sure what my mistake might be. Please find the bug I filed on the Ceph tracker here <https://tracker.ceph.com/issues/48026> where I've provided a mon log file with log level 20. Kind regards, Lalit Maganti

1 0

bluefs mount failed(crash) after a long time
by Elians Wan 30 Oct '20

30 Oct '20

Anyone can help? Bluefs mount failed after a long time The error message: 2020-10-30 05:33:54.906725 7f1ad73f5e00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-30/block size 7.28TiB 2020-10-30 05:33:54.906758 7f1ad73f5e00 1 bluefs mount 2020-10-30 06:00:32.881850 7f1ad73f5e00 -1 *** Caught signal (Segmentation fault) ** in thread 7f1ad73f5e00 thread_name:ceph-osd ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0xaa2044) [0x5570d12af044] 2: (()+0x11390) [0x7f1ad56d2390] 3: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xad4) [0x5570d125ea34] 4: (BlueFS::_replay(bool)+0x409) [0x5570d1267599] 5: (BlueFS::mount()+0x209) [0x5570d126b659] 6: (BlueStore::_open_db(bool)+0x169c) [0x5570d117acdc] 7: (BlueStore::_mount(bool)+0x3ad) [0x5570d11aeded] 8: (OSD::init()+0x3e2) [0x5570d0d00f12] 9: (main()+0x2f0a) [0x5570d0c0a0ca] 10: (__libc_start_main()+0xf0) [0x7f1ad4658830] 11: (_start()+0x29) [0x5570d0c97329] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2 1