ceph pool size 1 for (temporary and expendable data) still using 2X storage?
Hey Ceph Users!
With all the buzz around chia coin, I want to dedicate a few TB to
storage mining, really just to play with the chia CLI tool, and learn
how it all works.
At the whole concept is about dedicating disk space for large
calculation outputs, the data is meaningless.
For this reason, I am hoping to use a pool with size 1, min_size 1,
and did set up the same.
However, as a proxmox user, I noticed that this pool appears to still
use 2X storage space, or at a minimum, the pool's maximum size is
limited to 50% of total storage space (not that I plan on maxing out
my storage for this.)
I suspect there is a novice-user failsafe which ensures foolishly
configured size=1 is automatically treated as size=2...
Can anyone point me towards how best to leverage my ceph cluster to
store expendable data at size=1 without wasting x2 actual disk space?
My cluster is perfectly balanced, so I am reluctant to pull an osd
out, generally don't have any other disks on hand, and don't plan to
spend money on additional storage for this endeavour. I do want to
ensure I am not wasting more space than I am expecting though.
(Small hobby cluster, if it matters)
Josh
Hi Reed,
Thankyou so much for the input and support. We have tried using the
variable suggested by you, but could not see any impact on the current
system.
*"ceph fs set cephfs allow_standby_replay true " *it did not create any *impact
in the failover time*
Furthermore we have tried more scenarios that we tested using our test :
*scenario 1:*
[image: image.png]
- In this we have tried to see the logs at the new node on which the mds
will failover to, i.e in this case if we reboot cephnode2 so new active MDS
will be Cephnode1. Checking logs for cephnode1 in two scenarios:
- 1. *normal reboot of Cephnode2 by keeping the I/O operation in
progress,*
- we see that log at cephnode1 instantiates immediately and then wait
for sometime (around 15 seconds for some beacon time) + some
additional 6-7
seconds during which it activated MDS on cephnode1 and resumes I/O. Refer
logs as :
- 2021-04-29T15:49:42.480+0530 7fa747690700 1 mds.cephnode1 Updating
MDS map to version 505 from mon.2
2021-04-29T15:49:42.482+0530 7fa747690700 1 mds.0.505 handle_mds_map
i am now mds.0.505
2021-04-29T15:49:42.482+0530 7fa747690700 1 mds.0.505 handle_mds_map
state change up:boot --> up:replay
2021-04-29T15:49:42.482+0530 7fa747690700 1 mds.0.505 replay_start
2021-04-29T15:49:42.482+0530 7fa747690700 1 mds.0.505 recovery set
is
2021-04-29T15:49:42.482+0530 7fa747690700 1 mds.0.505 waiting for
osdmap 486 (which blacklists prior instance)
2021-04-29T15:49:55.686+0530 7fa74568c700 1 mds.beacon.cephnode1 MDS
connection to Monitors appears to be laggy; 15.9769s since last
acked beacon
2021-04-29T15:49:55.686+0530 7fa74568c700 1 mds.0.505 skipping
upkeep work because connection to Monitors appears laggy
2021-04-29T15:49:57.533+0530 7fa749e95700 0 mds.beacon.cephnode1
MDS is no longer laggy
2021-04-29T15:49:59.599+0530 7fa740e83700 0 mds.0.cache creating
system inode with ino:0x100
2021-04-29T15:49:59.599+0530 7fa740e83700 0 mds.0.cache creating
system inode with ino:0x1
2021-04-29T15:50:00.456+0530 7fa73f680700 1 mds.0.505 Finished
replaying journal
2021-04-29T15:50:00.456+0530 7fa73f680700 1 mds.0.505 making mds
journal writeable
2021-04-29T15:50:00.959+0530 7fa747690700 1 mds.cephnode1 Updating
MDS map to version 506 from mon.2
2021-04-29T15:50:00.959+0530 7fa747690700 1 mds.0.505 handle_mds_map
i am now mds.0.505
2021-04-29T15:50:00.959+0530 7fa747690700 1 mds.0.505 handle_mds_map
state change up:replay --> up:reconnect
2021-04-29T15:50:00.959+0530 7fa747690700 1 mds.0.505 reconnect_start
2021-04-29T15:50:00.959+0530 7fa747690700 1 mds.0.505 reopen_log
2021-04-29T15:50:00.959+0530 7fa747690700 1 mds.0.server
reconnect_clients -- 2 sessions
2021-04-29T15:50:00.964+0530 7fa747690700 0 log_channel(cluster) log
[DBG] : reconnect by client.6892 v1:10.0.4.96:0/1646469259 after
0.00499997
2021-04-29T15:50:00.972+0530 7fa747690700 0 log_channel(cluster) log
[DBG] : reconnect by client.6990 v1:10.0.4.115:0/2776266880 after
0.0129999
2021-04-29T15:50:00.972+0530 7fa747690700 1 mds.0.505 reconnect_done
2021-04-29T15:50:02.005+0530 7fa747690700 1 mds.cephnode1 Updating
MDS map to version 507 from mon.2
2021-04-29T15:50:02.005+0530 7fa747690700 1 mds.0.505 handle_mds_map
i am now mds.0.505
2021-04-29T15:50:02.005+0530 7fa747690700 1 mds.0.505 handle_mds_map
state change up:reconnect --> up:rejoin
2021-04-29T15:50:02.005+0530 7fa747690700 1 mds.0.505 rejoin_start
2021-04-29T15:50:02.008+0530 7fa747690700 1 mds.0.505
rejoin_joint_start
2021-04-29T15:50:02.040+0530 7fa740e83700 1 mds.0.505 rejoin_done
2021-04-29T15:50:03.050+0530 7fa747690700 1 mds.cephnode1 Updating
MDS map to version 508 from mon.2
2021-04-29T15:50:03.050+0530 7fa747690700 1 mds.0.505 handle_mds_map
i am now mds.0.505
2021-04-29T15:50:03.050+0530 7fa747690700 1 mds.0.505 handle_mds_map
state change up:rejoin --> up:clientreplay
2021-04-29T15:50:03.050+0530 7fa747690700 1 mds.0.505 recovery_done
-- successful recovery!
2021-04-29T15:50:03.050+0530 7fa747690700 1 mds.0.505
clientreplay_start
2021-04-29T15:50:03.094+0530 7fa740e83700 1 mds.0.505
clientreplay_done
2021-04-29T15:50:04.081+0530 7fa747690700 1 mds.cephnode1 Updating
MDS map to version 509 from mon.2
2021-04-29T15:50:04.081+0530 7fa747690700 1 mds.0.505 handle_mds_map
i am now mds.0.505
2021-04-29T15:50:04.081+0530 7fa747690700 1 mds.0.505 handle_mds_map
state change up:clientreplay --> up:active
2021-04-29T15:50:04.081+0530 7fa747690700 1 mds.0.505 active_start
2021-04-29T15:50:04.085+0530 7fa747690700 1 mds.0.505 cluster
recovered.
- *hard reset/power-off of Cephnode2 by keeping the I/O operation in
progress:*
- In this case we see that the system logs at cephnode 1(on which new
MDS will be activated) gets activated after 15+ seconds of power-off.
- Time at which power-off was it : 2021-04-29-16-17-37
- *Time at which the logs started to show in cephnode 1* (refer
logs) i.e log started nearly after 15 seconds of hardware reset:
- 2021-04-29T16:17:51.983+0530 7f5ba3a38700 1 mds.cephnode1 Updating
MDS map to version 518 from mon.0
2021-04-29T16:17:51.984+0530 7f5ba3a38700 1 mds.0.518
handle_mds_map i am now mds.0.518
2021-04-29T16:17:51.984+0530 7f5ba3a38700 1 mds.0.518
handle_mds_map state change up:boot --> up:replay
2021-04-29T16:17:51.984+0530 7f5ba3a38700 1 mds.0.518
replay_start
2021-04-29T16:17:51.984+0530 7f5ba3a38700 1 mds.0.518
recovery set is
2021-04-29T16:17:51.984+0530 7f5ba3a38700 1 mds.0.518 waiting
for osdmap 504 (which blacklists prior instance)
2021-04-29T16:17:54.044+0530 7f5b9ca2a700 0 mds.0.cache
creating system inode with ino:0x100
2021-04-29T16:17:54.045+0530 7f5b9ca2a700 0 mds.0.cache
creating system inode with ino:0x1
2021-04-29T16:17:55.025+0530 7f5b9ba28700 1 mds.0.518 Finished
replaying journal
2021-04-29T16:17:55.025+0530 7f5b9ba28700 1 mds.0.518 making
mds journal writeable
2021-04-29T16:17:56.060+0530 7f5ba3a38700 1 mds.cephnode1
Updating MDS map to version 519 from mon.0
2021-04-29T16:17:56.060+0530 7f5ba3a38700 1 mds.0.518
handle_mds_map i am now mds.0.518
2021-04-29T16:17:56.060+0530 7f5ba3a38700 1 mds.0.518
handle_mds_map state change up:replay --> up:reconnect
2021-04-29T16:17:56.060+0530 7f5ba3a38700 1 mds.0.518
reconnect_start
2021-04-29T16:17:56.060+0530 7f5ba3a38700 1 mds.0.518
reopen_log
2021-04-29T16:17:56.060+0530 7f5ba3a38700 1 mds.0.server
reconnect_clients -- 2 sessions
2021-04-29T16:17:56.068+0530 7f5ba3a38700 0
log_channel(cluster) log [DBG] : reconnect by client.6990 v1:
10.0.4.115:0/2776266880 after 0.00799994
2021-04-29T16:17:56.069+0530 7f5ba3a38700 0
log_channel(cluster) log [DBG] : reconnect by client.6892 v1:
10.0.4.96:0/1646469259 after 0.00899994
2021-04-29T16:17:56.069+0530 7f5ba3a38700 1 mds.0.518
reconnect_done
2021-04-29T16:17:57.099+0530 7f5ba3a38700 1 mds.cephnode1
Updating MDS map to version 520 from mon.0
2021-04-29T16:17:57.099+0530 7f5ba3a38700 1 mds.0.518
handle_mds_map i am now mds.0.518
2021-04-29T16:17:57.099+0530 7f5ba3a38700 1 mds.0.518
handle_mds_map state change up:reconnect --> up:rejoin
2021-04-29T16:17:57.099+0530 7f5ba3a38700 1 mds.0.518
rejoin_start
2021-04-29T16:17:57.103+0530 7f5ba3a38700 1 mds.0.518
rejoin_joint_start
2021-04-29T16:17:57.472+0530 7f5b9d22b700 1 mds.0.518
rejoin_done
2021-04-29T16:17:58.138+0530 7f5ba3a38700 1 mds.cephnode1
Updating MDS map to version 521 from mon.0
2021-04-29T16:17:58.138+0530 7f5ba3a38700 1 mds.0.518
handle_mds_map i am now mds.0.518
2021-04-29T16:17:58.138+0530 7f5ba3a38700 1 mds.0.518
handle_mds_map state change up:rejoin --> up:clientreplay
2021-04-29T16:17:58.138+0530 7f5ba3a38700 1 mds.0.518
recovery_done -- successful recovery!
2021-04-29T16:17:58.138+0530 7f5ba3a38700 1 mds.0.518
clientreplay_start
2021-04-29T16:17:58.157+0530 7f5b9d22b700 1 mds.0.518
clientreplay_done
2021-04-29T16:17:59.178+0530 7f5ba3a38700 1 mds.cephnode1
Updating MDS map to version 522 from mon.0
2021-04-29T16:17:59.178+0530 7f5ba3a38700 1 mds.0.518
handle_mds_map i am now mds.0.518
2021-04-29T16:17:59.178+0530 7f5ba3a38700 1 mds.0.518
handle_mds_map state change up:clientreplay --> up:active
2021-04-29T16:17:59.178+0530 7f5ba3a38700 1 mds.0.518
active_start
2021-04-29T16:17:59.181+0530 7f5ba3a38700 1 mds.0.518 cluster
recovered.
*In both the test cases above* we saw some extra delay of* around 15
seconds* + 8-10 seconds. (total 21-25 seconds for failover in case of
power-off/reboot),
*Query:* Any specific config that may need to be tweaked/tried to reduce
this time for MDS to know that it has to activate and start the standby MDS
Node?)
*Scenario 2:*
- *Only stop MDS Daemon Service on Active Node*
- In this scenario when we only tried stopping systemctl service for
the MDS Node on Active Node, we have very good *reading of around 5-7
Seconds* for failover.
-
Deployment Mode CEPH MDS Setup Test Case I/O Resume Duration
(Seconds) Node affected
-
2 Node MDS Setupwith max_mds=1 Active-Standby MDS with Active Node
MDS Demon stop 5-7 cephnode 1
Please suggest/advice if we can try to configure to achieve minimal
failover duration in the first two scenarios.
Best Regards,
Lokendra
On Thu, Apr 29, 2021 at 1:47 AM Reed Dier <reed.dier(a)focusvq.com> wrote:
> I don't have anything of merit to add to this, but it would be an
> interesting addition to your testing to see if active+standby-replay makes
> any difference with test-case1.
>
> I don't think it would be applicable to any of the other use-cases, as a
> standby-replay MDS is bound to a single rank, meaning its bound to a single
> active MDS, and can't function as a standby for active:active.
>
> https://docs.ceph.com/en/latest/cephfs/standby/#configuring-standby-replay
>
>
> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/c…
>
> Good luck and look forward to hearing feedback/more results.
>
> Reed
>
> On Apr 27, 2021, at 8:40 AM, Lokendra Rathour <lokendrarathour(a)gmail.com>
> wrote:
>
> Hi Team,
> We have setup two Node Ceph Cluster using *Native Cephfs Driver* with *Details
> as:*
>
> - 3 Node / 2 Node MDS Cluster
> - 3 Node Monitor Quorum
> - 2 Node OSD
> - 2 Nodes for Manager
>
>
> Cephnode3 have only Mon and MDS (only for test case 4-7) rest two nodes
> i.e. cephnode1 and cephnode2 have (mgr,mds,mon,rgw)
>
>
> We have tested following failover scenarios for Native Cephfs Driver by
> mounting for any one sub-volume on a VM or client with continuous I/O
> operations(Directory creation after every 1 Second)*:*
>
> <image.png>
>
>
> In the table above we have few queries as:
>
> - Refer test case 2 and test case 7, both are similar test case with
> only difference in number of Ceph MDS with time for both the test cases is
> different. It should be zero. But time is coming as 17 seconds for testcase
> 7.
> - Is there any configurable parameter/any configuration which we need
> to make in the Ceph cluster to get the failover time reduced to few
> seconds?
>
> In current default deployment we are getting something around 35-40
> seconds.
>
>
>
>
>
>
> Best Regards,
>
> --
> ~ Lokendra
> www.inertiaspeaks.com
> www.inertiagroups.com
> skype: lokendrarathour
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
>
--
~ Lokendra
www.inertiaspeaks.comwww.inertiagroups.com
skype: lokendrarathour
Hey all!
rbd export images/ec8a7ff8-6609-4b7d-8bdd-fadcf3b7973e /root/foo.img
DOES NOT produce the target file
no matter if I use --pool --image format or the one above the target
file is not there.
Progress bar shows up and prints percentage. It ends up with exit 0
[root@controller-0 mnt]# rbd export --pool=images db8290c3-93fd-4a4e-
ad71-7c131070ad6f /mnt/cirros2.img
Exporting image: 100% complete...done.
[root@controller-0 mnt]# echo $?
0
Any idea what's going on here?
It's ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94)
luminous (stable)
Any hints will be much appreciated
best regards
Piotr
Hi,
On a daily basis, one of my monitors goes down
[root@cube ~]# ceph health detail
HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum rhel1.robeckert.us,story
[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)
daemon mon.cube on cube.robeckert.us is in error state
[WRN] MON_DOWN: 1/3 mons down, quorum rhel1.robeckert.us,story
mon.cube (rank 2) addr [v2:192.168.2.142:3300/0,v1:192.168.2.142:6789/0] is down (out of quorum)
[root@cube ~]# ceph --version
ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)
I have a script that will copy the mon data from another server and it restarts and runs well for a while.
It is always the same monitor, and when I look at the logs the only thing I really see is the cephadm log showing it down
2021-04-28 10:07:26,173 DEBUG Running command: /usr/bin/podman --version
2021-04-28 10:07:26,217 DEBUG /usr/bin/podman: stdout podman version 2.2.1
2021-04-28 10:07:26,222 DEBUG Running command: /usr/bin/podman inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867-osd.2
2021-04-28 10:07:26,326 DEBUG /usr/bin/podman: stdout fab17e5242eb4875e266df19ca89b596a2f2b1d470273a99ff71da2ae81eeb3c,docker.io/ceph/ceph:v15,5b724076c58f97872fc2f7701e8405ec809047d71528f79da452188daf2af72e,2021-04-26 17:13:15.54183375 -0400 EDT,
2021-04-28 10:07:26,328 DEBUG Running command: systemctl is-enabled ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867(a)mon.cube<mailto:ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867@mon.cube>
2021-04-28 10:07:26,334 DEBUG systemctl: stdout enabled
2021-04-28 10:07:26,335 DEBUG Running command: systemctl is-active ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867(a)mon.cube<mailto:ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867@mon.cube>
2021-04-28 10:07:26,340 DEBUG systemctl: stdout failed
2021-04-28 10:07:26,340 DEBUG Running command: /usr/bin/podman --version
2021-04-28 10:07:26,395 DEBUG /usr/bin/podman: stdout podman version 2.2.1
2021-04-28 10:07:26,402 DEBUG Running command: /usr/bin/podman inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867-mon.cube
2021-04-28 10:07:26,526 DEBUG /usr/bin/podman: stdout 04e7c673cbacf5160427b0c3eb2f0948b2f15d02c58bd1d9dd14f975a84cfc6f,docker.io/ceph/ceph:v15,5b724076c58f97872fc2f7701e8405ec809047d71528f79da452188daf2af72e,2021-04-28 08:54:57.614847512 -0400 EDT,
I don't know if it matters, but this server is an AMD 3600XT while my other two servers which have had no issues are intel based.
The root file system was originally on a SSD, and I switched to NVME, so I eliminated controller or drive issues. (I didn't see anything in dmesg anyway)
If someone could point me in the right direction on where to troubleshoot next, I would appreciate it.
Thanks,
Rob Eckert
Hello,
I cannot flatten an image, it always restarts with:
root@sm-node1.in.illusion.hu:~# rbd flatten vm-hdd/vm-104-disk-1
Image flatten: 28% complete...2021-04-29 10:50:27.373 7ff7caffd700 -1
librbd::operation::FlattenRequest: 0x7ff7c4009db0 should_complete:
encountered error: (85) Interrupted system call should be restarted
Image flatten: 26% complete...2021-04-29 10:50:33.053 7ff7caffd700 -1
librbd::operation::FlattenRequest: 0x7ff7c4008fc0 should_complete:
encountered error: (85) Interrupted system call should be restarted
Image flatten: 0% complete...2021-04-29 10:50:34.829 7ff7caffd700 -1
librbd::operation::FlattenRequest: 0x7ff7c445b470 should_complete:
encountered error: (85) Interrupted system call should be restarted
Image flatten: 39% complete...2021-04-29 10:50:42.081 7ff7caffd700 -1
librbd::operation::FlattenRequest: 0x7ff7c40324e0 should_complete:
encountered error: (85) Interrupted system call should be restarted
Image flatten: 0% complete...2021-04-29 10:50:43.897 7ff7caffd700 -1
librbd::operation::FlattenRequest: 0x7ff7c4018890 should_complete:
encountered error: (85) Interrupted system call should be restarted
Image flatten: 42% complete...2021-04-29 10:51:07.813 7ff7caffd700 -1
librbd::operation::FlattenRequest: 0x7ff7c402fe80 should_complete:
encountered error: (85) Interrupted system call should be restarted
Image flatten: 42% complete...2021-04-29 10:51:29.372 7ff7caffd700 -1
librbd::operation::FlattenRequest: 0x7ff7c40017c0 should_complete:
encountered error: (85) Interrupted system call should be restarted
root@sm-node1.in.illusion.hu:~# uname -a
Linux sm-node1.in.illusion.hu 5.4.106-1-pve #1 SMP PVE 5.4.106-1 (Fri,
19 Mar 2021 11:08:47 +0100) x86_64 GNU/Linux
root@sm-node1.in.illusion.hu:~# dpkg -l |grep ceph
ii ceph 14.2.20-pve1 amd64
distributed storage and file system
ii ceph-base 14.2.20-pve1 amd64
common ceph daemon libraries and management tools
ii ceph-common 14.2.20-pve1 amd64
common utilities to mount and interact with a ceph storage cluster
ii ceph-fuse 14.2.20-pve1 amd64
FUSE-based client for the Ceph distributed file system
ii ceph-mds 14.2.20-pve1 amd64
metadata server for the ceph distributed file system
ii ceph-mgr 14.2.20-pve1 amd64
manager for the ceph distributed storage system
ii ceph-mgr-dashboard 14.2.20-pve1
all dashboard plugin for ceph-mgr
ii ceph-mon 14.2.20-pve1 amd64
monitor server for the ceph storage system
ii ceph-osd 14.2.20-pve1 amd64 OSD
server for the ceph storage system
ii libcephfs2 14.2.20-pve1 amd64 Ceph
distributed file system client library
ii python-ceph-argparse 14.2.20-pve1
all Python 2 utility libraries for Ceph CLI
ii python-cephfs 14.2.20-pve1 amd64
Python 2 libraries for the Ceph libcephfs library
root@sm-node1.in.illusion.hu:~# dpkg -l |grep rbd
ii librbd1 14.2.20-pve1 amd64 RADOS
block device client library
ii python-rbd 14.2.20-pve1 amd64
Python 2 libraries for the Ceph librbd library
root@sm-node1.in.illusion.hu:~# modinfo rbd
filename: /lib/modules/5.4.106-1-pve/kernel/drivers/block/rbd.ko
license: GPL
description: RADOS Block Device (RBD) driver
author: Jeff Garzik <jeff(a)garzik.org>
author: Yehuda Sadeh <yehuda(a)hq.newdream.net>
author: Sage Weil <sage(a)newdream.net>
author: Alex Elder <elder(a)inktank.com>
srcversion: 7BA6FEE20249E416B2D09AB
depends: libceph
retpoline: Y
intree: Y
name: rbd
vermagic: 5.4.106-1-pve SMP mod_unload modversions
parm: single_major:Use a single major number for all rbd
devices (default: true) (bool)
Please give me advice, how can I produce more information to catch this.
Thank you,
i.
Hello everyone,
I am running ceph version 15.2.8 on Ubuntu servers. I am using bluestore osds with data on hdd and db and wal on ssd drives. Each ssd has been partitioned such that it holds 5 dbs and 5 wals. The ssd were were prepared a while back probably when I was running ceph 13.x. I have been gradually adding new osd drives as needed. Recently, I've tried to add more osds, which have failed to my surprise. Previously I've had no issues adding the drives. However, it seems that I can no longer do that with version 15.2.x
Here is what I get:
root@arh-ibstorage4-ib /home/andrei ceph-volume lvm prepare --bluestore --data /dev/sds --block.db /dev/ssd3/db5 --block.wal /dev/ssd3/wal5
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 6aeef34b-0724-4d20-a10b-197cab23e24d
Running command: /usr/sbin/vgcreate --force --yes ceph-1c7cef26-327a-4785-96b3-dcb1b97e8e2f /dev/sds
stderr: WARNING: PV /dev/sdp in VG ceph-bc7587b5-0112-4097-8c9f-4442e8ea5645 is using an old PV header, modify the VG to update.
stderr: WARNING: PV /dev/sdo in VG ceph-33eda27c-53ed-493e-87a8-39e1862da809 is using an old PV header, modify the VG to update.
stderr: WARNING: PV /dev/sdn in VG ssd2 is using an old PV header, modify the VG to update.
stderr: WARNING: PV /dev/sdm in VG ssd1 is using an old PV header, modify the VG to update.
stderr: WARNING: PV /dev/sdj in VG ceph-9d8da00c-f6b9-473f-b499-fa60d74b46c5 is using an old PV header, modify the VG to update.
stderr: WARNING: PV /dev/sdi in VG ceph-1603149e-1e50-4b86-a360-1372f4243603 is using an old PV header, modify the VG to update.
stderr: WARNING: PV /dev/sdh in VG ceph-a5f4416c-8e69-4a66-a884-1d1229785acb is using an old PV header, modify the VG to update.
stderr: WARNING: PV /dev/sde in VG ceph-aac71121-e308-4e25-ae95-ca51bca7aaff is using an old PV header, modify the VG to update.
stderr: WARNING: PV /dev/sdd in VG ceph-1e216580-c01b-42c5-a10f-293674a55c4c is using an old PV header, modify the VG to update.
stderr: WARNING: PV /dev/sdc in VG ceph-630f7716-3d05-41bb-92c9-25402e9bb264 is using an old PV header, modify the VG to update.
stderr: WARNING: PV /dev/sdb in VG ceph-a549c28d-9b06-46d5-8ba3-3bd99ff54f57 is using an old PV header, modify the VG to update.
stderr: WARNING: PV /dev/sda in VG ceph-70943bd0-de71-4651-a73d-c61bc624755f is using an old PV header, modify the VG to update.
stdout: Physical volume "/dev/sds" successfully created.
stdout: Volume group "ceph-1c7cef26-327a-4785-96b3-dcb1b97e8e2f" successfully created
Running command: /usr/sbin/lvcreate --yes -l 3814911 -n osd-block-6aeef34b-0724-4d20-a10b-197cab23e24d ceph-1c7cef26-327a-4785-96b3-dcb1b97e8e2f
stdout: Logical volume "osd-block-6aeef34b-0724-4d20-a10b-197cab23e24d" created.
--> blkid could not detect a PARTUUID for device: /dev/ssd3/wal5
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.15 --yes-i-really-mean-it
stderr: 2021-04-28T20:05:52.290+0100 7f76bbfa9700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc
/ceph/keyring.bin,: (2) No such file or directory
2021-04-28T20:05:52.290+0100 7f76bbfa9700 -1 AuthRegistry(0x7f76b4058e60) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyrin
g,/etc/ceph/keyring.bin,, disabling cephx
stderr: purged osd.15
--> RuntimeError: unable to use device
I have tried to find a solution, but wasn't able to resolve the problem? I am sure that I've previously added new volumes using the above command.
lvdisplay shows:
--- Logical volume ---
LV Path /dev/ssd3/wal5
LV Name wal5
VG Name ssd3
LV UUID WPQJs9-olAj-ACbU-qnEM-6ytu-aLMv-hAABYy
LV Write Access read/write
LV Creation host, time arh-ibstorage4-ib, 2020-07-29 23:45:17 +0100
LV Status available
# open 0
LV Size 1.00 GiB
Current LE 256
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:6
--- Logical volume ---
LV Path /dev/ssd3/db5
LV Name db5
VG Name ssd3
LV UUID FVT2Mm-a00P-eCoQ-FZAf-AulX-4q9r-PaDTC6
LV Write Access read/write
LV Creation host, time arh-ibstorage4-ib, 2020-07-29 23:46:01 +0100
LV Status available
# open 0
LV Size 177.00 GiB
Current LE 45312
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:11
How do I resolve the errors and create the new osd?
Cheers
Andrei
Hello,
Last week I upgraded my production cluster to Pacific. the cluster was
healthy until a few hours ago.
When scrub run 4hrs ago left the cluster in an inconsistent state. Then
issued the command ceph pg repair 7.182 to try to repair the cluster but
ended with active+recovery_unfound+degraded
All OSDs are up and all running bluestore with replication of 3 and minimum
size of 2. I have restarted all OSD but still not helping.
Any recommendations on how to recover the cluster safely?
I have attached result of ceph pg 7.182 query
ceph health detail
HEALTH_ERR 1/2459601 objects unfound (0.000%); Possible data damage: 1 pg
recovery_unfound; Degraded data redundancy: 3/7045706 objects degraded
(0.000%), 1 pg degraded
[WRN] OBJECT_UNFOUND: 1/2459601 objects unfound (0.000%)
pg 7.182 has 1 unfound objects
[ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound
pg 7.182 is active+recovery_unfound+degraded, acting [15,1,11], 1
unfound
[WRN] PG_DEGRADED: Degraded data redundancy: 3/7045706 objects degraded
(0.000%), 1 pg degraded
pg 7.182 is active+recovery_unfound+degraded, acting [15,1,11], 1
unfound
ceph -w
cluster:
id: 4b9f6959-fead-4ada-ac58-de5d7b149286
health: HEALTH_ERR
1/2459586 objects unfound (0.000%)
Possible data damage: 1 pg recovery_unfound
Degraded data redundancy: 3/7045661 objects degraded (0.000%),
1 pg degraded
services:
mon: 3 daemons, quorum mon-a,mon-b,mon-c (age 38m)
mgr: mon-a(active, since 38m)
osd: 46 osds: 46 up (since 25m), 46 in (since 3w)
data:
pools: 4 pools, 705 pgs
objects: 2.46M objects, 9.1 TiB
usage: 24 TiB used, 95 TiB / 119 TiB avail
pgs: 3/7045661 objects degraded (0.000%)
1/2459586 objects unfound (0.000%)
701 active+clean
3 active+clean+scrubbing+deep
1 active+recovery_unfound+degraded
ceph pg 7.182 list_unfound
{
"num_missing": 1,
"num_unfound": 1,
"objects": [
{
"oid": {
"oid": "rbd_data.2f18f2a67fad72.000000000002021a",
"key": "",
"snapid": -2,
"hash": 3951004034,
"max": 0,
"pool": 7,
"namespace": ""
},
"need": "184249'118613008",
"have": "0'0",
"flags": "none",
"clean_regions": "clean_offsets: [], clean_omap: 0, new_object:
1",
"locations": []
}
],
"state": "NotRecovering",
"available_might_have_unfound": true,
"might_have_unfound": [],
"more": false
}
Hi,
I have a bucket that has versioning enabled and I am trying to remove
it. This is not possible, because the bucket is not empty:
mark@tuxis:~$ aws --endpoint=https://nl.dadup.eu s3api delete-bucket
--bucket syslog_tuxis_net
An error occurred (BucketNotEmpty) when calling the DeleteBucket
operation: Unknown
It took me a while to determine that `curator` had enabled bucket
versioning, but after I did I removed all versions.
But, when trying to delete the bucket, it still doesn't work because
there are DeleteMarkers, eg:
{
"Owner": {
"DisplayName": "Syslog Backup",
"ID": "DB0220$syslog_backup"
},
"Key": "incompatible-snapshots",
"VersionId": "noeJtvBpV5HINGQTJeXEq5mzlzsWneg",
"IsLatest": true,
"LastModified": "2021-03-22T16:35:18.298Z"
},
I cannot download that object:
mark@tuxis:~$ aws --endpoint=https://nl.dadup.eu s3api get-object
--bucket syslog_tuxis_net --key incompatible-snapshots /tmp/foobar
An error occurred (NoSuchKey) when calling the GetObject operation:
Unknown
Nor can I delete the object:
mark@tuxis:~$ aws --endpoint=https://nl.dadup.eu s3api delete-object
--bucket syslog_tuxis_net --key incompatible-snapshots
So, according to
https://ceph.io/planet/on-ceph-rgw-s3-object-versioning/#on-delete-marker
I should be able to delete that DeleteMarker with this command:
mark@tuxis:~$ aws --endpoint=https://nl.dadup.eu s3api delete-object
--bucket syslog_tuxis_net --key incompatible-snapshots --version-id
noeJtvBpV5HINGQTJeXEq5mzlzsWneg
But that command does not give any output, and it does not delete the
marker either.
So I'm stuck with that bucket which I would like to remove without
abusing radosgw-admin.
This cluster is running 12.2.13 with civetweb rgw's behind a haproxy
setup. All is working fine, except for this versioning bucket. Can
anywone point me in the right direction to remove this bucket as a
normal user?
Thanks!
--
Mark Schouten | Tuxis B.V.
KvK: 74698818 | http://www.tuxis.nl/
T: +31 318 200208 | info(a)tuxis.nl
Hello,
We’ve got some issues when uploading s3 objects with a double slash // in the name, and was wondering if anyone else has observed this issue with uploading objects to the radosgw?
When connecting to the cluster to upload an object with the key ‘test/my//bucket’ the request returns with a 403 (SignatureDoesNotMatch) error.
Wondering if anyone else has observed this behavior and has any workarounds to work with double slashes in the object key name.
- Gavin
hello all,
I faced an incident for one of my very important rbd volumes with 5TB data, which is managed by OpenStack.
I was about to increase the volume size live but I shrinked the volume unintentionally by running a wrong command of "virsh qemu-monitor-command". then I realized it and expand it again, but obviously I lost my data. can you help me or give me hints on how I can recover data of this rbd volume?
unfortunately, I don't have any backup for this part of data, and it's really important (I know I made a big mistake). also, I can't stop the cluster since it's under heavy production load.
It seems qemu has been using the old set of APIs that allows shrinking by default without any warnings even in the latest version. (All other standard ways of resizing volume, does not allow shrinking)
* Ceph: https://sourcegraph.com/github.com/ceph/ceph@luminous/-/blob/src/librbd/lib…
* Qemu: https://sourcegraph.com/github.com/qemu/qemu/-/blob/block/rbd.c#L832
* Libvirt: https://sourcegraph.com/github.com/libvirt/libvirt/-/blob/src/storage/stora…
