- ceph-users - lists.ceph.io

Ceph Object Gateway and lua scripts
by Thomas Bennett 11 Apr '23

11 Apr '23

Hi, We're currently testing out lua scripting in the Ceph Object Gateway (Radosgw). Ceph version: 17.2.5 We've tried a simple experiment with the simple lua script which is based on the documentation (see fixed width text below). However, the issue we're having is that we can't find the log messages anywhere. We've searched the entire jourrnalctl database as well as raised the debug level on the radosgw by setting debug_rgw to 20 on the running daemon. Any help welcome :) function print_object(msg, object) RGWDebugLog(" Title: " .. msg) RGWDebugLog(" Name: " .. object.Name) RGWDebugLog(" Instance: " .. object.Instance) RGWDebugLog(" Id: " .. object.Id) RGWDebugLog(" Size: " .. object.Size) RGWDebugLog(" MTime: " .. object.MTime) end RGWDebugLog("This is a log message!") Request.Log() if Request.CopyFrom then print_object("copy from", Request.CopyFrom.Object) if Request.CopyFrom.Object then print_object("copy from-object" ,Request.CopyFrom.Object) end end if Request.Object then print_object("Object" ,Request.Object) end Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.

2 3

RGW don't use .rgw.root multisite configuration
by Guillaume Morin 11 Apr '23

11 Apr '23

Hello, for my multisite configuration, i create and use an root pool other than .rgw.root to store realm and zone configuration with the follow option: rgw_realm_root_pool=myzone.rgw.root rgw_zonegroup_root_pool=myzone.rgw.root rgw_zone_root_pool=myzone.rgw.root But , i can see during the syncronize with my secondary zone (real only) , radosgw need to read and write period in .rgw.root. Is there an option or other to avoid read and write on the root pool .rgw.root ? Is it mandatory ? Pacific regards Guillaume

3 2

Announcing go-ceph v0.21.0
by John Mulligan 11 Apr '23

11 Apr '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.21.0 Changes include additions to the rbd, cephfs, and cephfs/admin packages. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0

Why is my cephfs almostfull?
by Jorge Garcia 11 Apr '23

11 Apr '23

We have a ceph cluster with a cephfs filesystem that we use mostly for backups. When I do a "ceph -s" or a "ceph df", it reports lots of space: data: pools: 3 pools, 4104 pgs objects: 1.09 G objects, 944 TiB usage: 1.5 PiB used, 1.0 PiB / 2.5 PiB avail GLOBAL: SIZE AVAIL RAW USED %RAW USED 2.5 PiB 1.0 PiB 1.5 PiB 59.76 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS cephfs_data 2 944 TiB 87.63 133 TiB 880988429 cephfs_metadata 3 128 MiB 0 62 TiB 206535313 .rgw.root 4 0 B 0 62 TiB 0 The whole thing consists of 2 pools: metadata (regular default replication) and data (erasure k:5 m:2). The global raw space reports 2.5 PiB total, with 1.0 PiB still available. But, when the ceph filesystem is mounted, it only reports 1.1 PB total, and the filesystem is almost full: Filesystem Size Used Avail Use% Mounted on x.x.x.x:yyyy:/ 1.1P 944T 134T 88% /backups So, where is the rest of my space? Or what am I missing? Thanks!

3 2

Disks are filling up even if there is not a single placement group on them
by Michal Strnad 11 Apr '23

11 Apr '23

Hi all, Our SSD disks are filling up, even if there is not a single placement group on them. How is that possible please? ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1997 ssd 1.15269 1.00000 1.2 TiB 943 GiB 942 GiB 6 KiB 1.5 GiB 237 GiB 79.90 1.71 0 up Cluster uses Pacific 16.2.7. Thank you Michal

3 4

ceph.v17 multi-mds ephemeral directory pinning: cannot set or retrieve extended attribute
by Ulrich Pralle 10 Apr '23

10 Apr '23

Hi, we are using ceph version 17.2.5 on Ubuntu 22.04.1 LTS. We deployed multi-mds (max_mds=4, plus standby-replay mds). Currently we statically directory-pinned our user home directories (~50k). The cephfs' root directory is pinned to '-1', ./homes is pinned to "0". All user home directories below ./homes/ are pinned to -1, 1, 2, or 3 depending on a simple hash algorithm. Cephfs is provided to our users as samba/cifs (clustered samba,ctdb). We want to try ephemeral directory pinning. We can successfully set the extended attribute "ceph.dir.pin.distributed" with setfattr(1), but cannot retrieve its setting afterwards.: # setfattr -n ceph.dir.pin.distributed -v 1 ./units # getfattr -n ceph.dir.pin.distributed ./units ./units: ceph.dir.pin.distributed: No such attribute strace setfattr reports success on setxattr setxattr("./units", "ceph.dir.pin.distributed", "1", 1, 0) = 0 strace getfattr reports lstat("./units", {st_mode=S_IFDIR|0751, st_size=1, ...}) = 0 getxattr("./units", "ceph.dir.pin.distributed", NULL, 0) = -1 ENODATA (No data available) The file system is mounted rw,noatime,,name=<omitted>,mds_namespace=<omitted>.acl,recover_session=clean. The cephfs mds caps are "allow rwps". "./units" has a ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=fs_data_units" Ubuntu's setfattr is version 2.4.48. Defining other cephfs extend attributes (like ceph.dir.pin, ceph.quota.max_bytes, etc.) works as expected. What are we missing? Should we clear all static directory pinnings in advance? Are there any experience with ephemeral directory pinning? Or should one refrain from multi-mds at all? Kind regards Uli Pralle

2 1

v17.2.6 Quincy released
by Yuri Weinstein 10 Apr '23

10 Apr '23

We're happy to announce the 6th backport release in the Quincy series. https://ceph.io/en/news/blog/2023/v17-2-6-quincy-released/ Notable Changes --------------- * `ceph mgr dump` command now outputs `last_failure_osd_epoch` and `active_clients` fields at the top level. Previously, these fields were output under `always_on_modules` field. * telemetry: Added new metrics to the 'basic' channel to report per-pool bluestore compression metrics. See a sample report with `ceph telemetry preview`. Opt-in with `ceph telemetry on`. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-17.2.6.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: d7ff0d10654d2280e08f1ab989c7cdf3064446a5

1 0

quincy v17.2.6 QE Validation status
by Yuri Weinstein 07 Apr '23

07 Apr '23

Details of this release are summarized here: https://tracker.ceph.com/issues/59070#note-1 Release Notes - TBD The reruns were in the queue for 4 days because of some slowness issues. The core team (Neha, Radek, Laura, and others) are trying to narrow down the root cause. Seeking approvals/reviews for: rados - Neha, Radek, Travis, Ernesto, Adam King (we still have to test and merge at least one PR https://github.com/ceph/ceph/pull/50575 for the core) rgw - Casey fs - Venky (the fs suite has an unusually high amount of failed jobs, any reason to suspect it in the observed slowness?) orch - Adam King rbd - Ilya krbd - Ilya upgrade/octopus-x - Laura is looking into failures upgrade/pacific-x - Laura is looking into failures upgrade/quincy-p2p - Laura is looking into failures client-upgrade-octopus-quincy-quincy - missing packages, Adam Kraitman is looking into it powercycle - Brad ceph-volume - needs a rerun on merged https://github.com/ceph/ceph-ansible/pull/7409 Please reply to this email with approval and/or trackers of known issues/PRs to address them. Also, share any findings or hypnosis about the slowness in the execution of the suite. Josh, Neha - gibba and LRC upgrades pending major suites approvals. RC release - pending major suites approvals. Thx YuriW

14 27

Cephadm - Error ENOENT: Module not found
by elia.oggian＠cscs.ch 06 Apr '23

06 Apr '23

Hello, After a successful upgrade of a Ceph cluster from 16.2.7 to 16.2.11, I needed to downgrade it back to 16.2.7 as I found an issue with the new version. I expected that running the downgrade with:`ceph orch upgrade start --ceph-version 16.2.7` should have worked fine. However, it blocked right after the downgrade of the first MGR daemon. In fact, the downgraded daemon is not able to use the cephadm module anymore. Any `ceph orch` command fails with the following error: ``` $ ceph orch ps Error ENOENT: Module not found ``` And the downgrade process is therefore blocked. These are the logs of the MGR when issuing the command: ``` Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.557+0000 7f828fe8c700 0 log_channel(audit) log [DBG] : from='client.3136173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.list_daemons(*(None, None), **{'daemon_id': None, 'host': None, 'refresh': False}) Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm' Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.get_feature_set(*(), **{}) Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm' Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 mgr.server reply reply (2) No such file or directory Module not found ``` Other interesting MGR logs are: ``` 2023-03-28T11:05:59.519+0000 7fcd16314700 4 mgr get_store get_store key: mgr/cephadm/upgrade_state 2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Failed to construct class in 'cephadm' 2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Traceback (most recent call last): e "/usr/share/ceph/mgr/cephadm/module.py", line 450, in __init__ elf.upgrade = CephadmUpgrade(self) e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 111, in __init__ elf.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 92, in from_json eturn cls(**c) rror: __init__() got an unexpected keyword argument 'daemon_types' 2023-03-28T11:05:59.521+0000 7fcd16314700 -1 mgr operator() Failed to run module in active mode ('cephadm') ``` Which seem to relate to the new feature of staggered upgrades. Please note that before, everything was working fine with version 16.2.7. I am currently stuck in this situation with only one MGR daemon on version 16.2.11 which is the only one still working fine: ``` [root@astano01 ~]# ceph orch ps | grep mgr mgr.astano02.mzmewn astano02 *:8443,9283 running (5d) 43s ago 2y 455M - 16.2.11 7a63bce27215 e2d7806acf16 mgr.astano03.qtzccn astano03 *:8443,9283 running (3m) 22s ago 95m 383M - 16.2.7 463ec4b1fdc0 cc0d88864fa1 ``` Does anyone already faced this issue or knows how can I make the 16.2.7 MGR load the cephadm module correctly? Thanks in advance for any help!

2 2

Misplaced objects greater than 100%
by Johan Hattne 06 Apr '23

06 Apr '23

Dear all; Up until a few hours ago, I had a seemingly normally-behaving cluster (Quincy, 17.2.5) with 36 OSDs, evenly distributed across 3 of its 6 nodes. The cluster is only used for CephFS and the only non-standard configuration I can think of is that I had 2 active MDSs, but only 1 standby. I had also doubled mds_cache_memory limit to 8 GB (all OSD hosts have 256 G of RAM) at some point in the past. Then I rebooted one of the OSD nodes. The rebooted node held one of the active MDSs. Now the node is back up: ceph -s says the cluster is healthy, but all PGs are in a active+clean+remapped state and 166.67% of the objects are misplaced (dashboard: -66.66% healthy). The data pool is a threefold replica with 5.4M object, the number of misplaced objects is reported as 27087410/16252446. The denominator in the ratio makes sense to me (16.2M / 3 = 5.4M), but the numerator does not. I also note that the ratio is *exactly* 5 / 3. The filesystem is still mounted and appears to be usable, but df reports it as 100% full; I suspect it would say 167% but that is capped somewhere. Any ideas about what is going on? Any suggestions for recovery? // Best wishes; Johan

3 6