With the Reef dev cycle closing, it's time to think about S and future
releases.
There are a bunch of options for S already, add a +1 or a new option to
this etherpad, and we'll see what has the most votes next week:
https://pad.ceph.com/p/s
Josh
Hi,
We're currently testing out lua scripting in the Ceph Object Gateway
(Radosgw).
Ceph version: 17.2.5
We've tried a simple experiment with the simple lua script which is based
on the documentation (see fixed width text below).
However, the issue we're having is that we can't find the log messages
anywhere. We've searched the entire jourrnalctl database as well as raised
the debug level on the radosgw by setting debug_rgw to 20 on the running
daemon.
Any help welcome :)
function print_object(msg, object)
RGWDebugLog(" Title: " .. msg)
RGWDebugLog(" Name: " .. object.Name)
RGWDebugLog(" Instance: " .. object.Instance)
RGWDebugLog(" Id: " .. object.Id)
RGWDebugLog(" Size: " .. object.Size)
RGWDebugLog(" MTime: " .. object.MTime)
end
RGWDebugLog("This is a log message!")
Request.Log()
if Request.CopyFrom then
print_object("copy from", Request.CopyFrom.Object)
if Request.CopyFrom.Object then
print_object("copy from-object" ,Request.CopyFrom.Object)
end
end
if Request.Object then
print_object("Object" ,Request.Object)
end
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.
Hello, for my multisite configuration, i create and use an root pool other than .rgw.root to store realm and zone configuration with the follow option:
rgw_realm_root_pool=myzone.rgw.root
rgw_zonegroup_root_pool=myzone.rgw.root
rgw_zone_root_pool=myzone.rgw.root
But , i can see during the syncronize with my secondary zone (real only) , radosgw need to read and write period in .rgw.root.
Is there an option or other to avoid read and write on the root pool .rgw.root ? Is it mandatory ?
Pacific
regards
Guillaume
We are happy to announce another release of the go-ceph API library.
This is a regular release following our every-two-months release
cadence.
https://github.com/ceph/go-ceph/releases/tag/v0.21.0
Changes include additions to the rbd, cephfs, and cephfs/admin packages.
More details are available at the link above.
The library includes bindings that aim to play a similar role to the
"pybind" python bindings in the ceph tree but for the Go language. The
library also includes additional APIs that can be used to administer
cephfs, rbd, and rgw subsystems.
There are already a few consumers of this library in the wild,
including the ceph-csi project.
--
John Mulligan
phlogistonjohn(a)asynchrono.us
jmulligan(a)redhat.com
We have a ceph cluster with a cephfs filesystem that we use mostly for
backups. When I do a "ceph -s" or a "ceph df", it reports lots of space:
data:
pools: 3 pools, 4104 pgs
objects: 1.09 G objects, 944 TiB
usage: 1.5 PiB used, 1.0 PiB / 2.5 PiB avail
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
2.5 PiB 1.0 PiB 1.5 PiB 59.76
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
cephfs_data 2 944 TiB 87.63 133 TiB 880988429
cephfs_metadata 3 128 MiB 0 62 TiB 206535313
.rgw.root 4 0 B 0 62
TiB 0
The whole thing consists of 2 pools: metadata (regular default
replication) and data (erasure k:5 m:2). The global raw space reports
2.5 PiB total, with 1.0 PiB still available. But, when the ceph
filesystem is mounted, it only reports 1.1 PB total, and the filesystem
is almost full:
Filesystem Size Used Avail Use% Mounted on
x.x.x.x:yyyy:/ 1.1P 944T 134T 88% /backups
So, where is the rest of my space? Or what am I missing?
Thanks!
Hi all,
Our SSD disks are filling up, even if there is not a single placement
group on them. How is that possible please?
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP
META AVAIL %USE VAR PGS STATUS
1997 ssd 1.15269 1.00000 1.2 TiB 943 GiB 942 GiB 6 KiB 1.5
GiB 237 GiB 79.90 1.71 0 up
Cluster uses Pacific 16.2.7.
Thank you
Michal
Hi,
we are using ceph version 17.2.5 on Ubuntu 22.04.1 LTS.
We deployed multi-mds (max_mds=4, plus standby-replay mds).
Currently we statically directory-pinned our user home directories (~50k).
The cephfs' root directory is pinned to '-1', ./homes is pinned to "0".
All user home directories below ./homes/ are pinned to -1, 1, 2, or 3
depending on a simple hash algorithm.
Cephfs is provided to our users as samba/cifs (clustered samba,ctdb).
We want to try ephemeral directory pinning.
We can successfully set the extended attribute
"ceph.dir.pin.distributed" with setfattr(1), but cannot retrieve its
setting afterwards.:
# setfattr -n ceph.dir.pin.distributed -v 1 ./units
# getfattr -n ceph.dir.pin.distributed ./units
./units: ceph.dir.pin.distributed: No such attribute
strace setfattr reports success on setxattr
setxattr("./units", "ceph.dir.pin.distributed", "1", 1, 0) = 0
strace getfattr reports
lstat("./units", {st_mode=S_IFDIR|0751, st_size=1, ...}) = 0
getxattr("./units", "ceph.dir.pin.distributed", NULL, 0) = -1 ENODATA
(No data available)
The file system is mounted
rw,noatime,,name=<omitted>,mds_namespace=<omitted>.acl,recover_session=clean.
The cephfs mds caps are "allow rwps".
"./units" has a ceph.dir.layout="stripe_unit=4194304 stripe_count=1
object_size=4194304 pool=fs_data_units"
Ubuntu's setfattr is version 2.4.48.
Defining other cephfs extend attributes (like ceph.dir.pin,
ceph.quota.max_bytes, etc.) works as expected.
What are we missing?
Should we clear all static directory pinnings in advance?
Are there any experience with ephemeral directory pinning?
Or should one refrain from multi-mds at all?
Kind regards
Uli Pralle
Details of this release are summarized here:
https://tracker.ceph.com/issues/59070#note-1
Release Notes - TBD
The reruns were in the queue for 4 days because of some slowness issues.
The core team (Neha, Radek, Laura, and others) are trying to narrow
down the root cause.
Seeking approvals/reviews for:
rados - Neha, Radek, Travis, Ernesto, Adam King (we still have to test
and merge at least one PR https://github.com/ceph/ceph/pull/50575 for
the core)
rgw - Casey
fs - Venky (the fs suite has an unusually high amount of failed jobs,
any reason to suspect it in the observed slowness?)
orch - Adam King
rbd - Ilya
krbd - Ilya
upgrade/octopus-x - Laura is looking into failures
upgrade/pacific-x - Laura is looking into failures
upgrade/quincy-p2p - Laura is looking into failures
client-upgrade-octopus-quincy-quincy - missing packages, Adam Kraitman
is looking into it
powercycle - Brad
ceph-volume - needs a rerun on merged
https://github.com/ceph/ceph-ansible/pull/7409
Please reply to this email with approval and/or trackers of known
issues/PRs to address them.
Also, share any findings or hypnosis about the slowness in the
execution of the suite.
Josh, Neha - gibba and LRC upgrades pending major suites approvals.
RC release - pending major suites approvals.
Thx
YuriW
Hello,
After a successful upgrade of a Ceph cluster from 16.2.7 to 16.2.11, I needed to downgrade it back to 16.2.7 as I found an issue with the new version.
I expected that running the downgrade with:`ceph orch upgrade start --ceph-version 16.2.7` should have worked fine. However, it blocked right after the downgrade of the first MGR daemon. In fact, the downgraded daemon is not able to use the cephadm module anymore. Any `ceph orch` command fails with the following error:
```
$ ceph orch ps
Error ENOENT: Module not found
```
And the downgrade process is therefore blocked.
These are the logs of the MGR when issuing the command:
```
Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.557+0000 7f828fe8c700 0 log_channel(audit) log [DBG] : from='client.3136173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch
Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.list_daemons(*(None, None), **{'daemon_id': None, 'host': None, 'refresh': False})
Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm'
Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.get_feature_set(*(), **{})
Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm'
Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 mgr.server reply reply (2) No such file or directory Module not found
```
Other interesting MGR logs are:
```
2023-03-28T11:05:59.519+0000 7fcd16314700 4 mgr get_store get_store key: mgr/cephadm/upgrade_state
2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Failed to construct class in 'cephadm'
2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Traceback (most recent call last):
e "/usr/share/ceph/mgr/cephadm/module.py", line 450, in __init__
elf.upgrade = CephadmUpgrade(self)
e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 111, in __init__
elf.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t))
e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 92, in from_json
eturn cls(**c)
rror: __init__() got an unexpected keyword argument 'daemon_types'
2023-03-28T11:05:59.521+0000 7fcd16314700 -1 mgr operator() Failed to run module in active mode ('cephadm')
```
Which seem to relate to the new feature of staggered upgrades.
Please note that before, everything was working fine with version 16.2.7.
I am currently stuck in this situation with only one MGR daemon on version 16.2.11 which is the only one still working fine:
```
[root@astano01 ~]# ceph orch ps | grep mgr
mgr.astano02.mzmewn astano02 *:8443,9283 running (5d) 43s ago 2y 455M - 16.2.11 7a63bce27215 e2d7806acf16
mgr.astano03.qtzccn astano03 *:8443,9283 running (3m) 22s ago 95m 383M - 16.2.7 463ec4b1fdc0 cc0d88864fa1
```
Does anyone already faced this issue or knows how can I make the 16.2.7 MGR load the cephadm module correctly?
Thanks in advance for any help!