I am seeing the following errors on an RGW multisite slave:
1. ERROR: failed to fetch mdlog info
2. failed to fetch local sync status: (5) Input/output error
Data seems to be replicating but not metadata. Does anyone have any
ideas on what may be wrong?
-----
# radosgw-admin sync status
realm 8f7fd3fd-f72d-411d-b06b-7b4b579f5f2f (prod)
zonegroup 60a2cb75-6978-46a3-b830-061c8be9dc75 (prod)
zone ffce148e-3b24-462d-98bf-8c212de31de5 (us-east-1)
2019-12-27 12:29:13.329597 7f71c4ec9dc0 0 meta sync: ERROR: failed to
fetch mdlog info
metadata sync syncing
full sync: 0/64 shards
failed to fetch local sync status: (5) Input/output error
data sync source: 7fe96e52-d6f7-4ad6-b66e-ecbbbffbc18e (us-east-2)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 12 shards
behind shards: [29,31,33,45,46,48,54,76,87,113,120,127]
oldest incremental change not applied:
2019-12-27 12:28:58.0.107159s
23 shards are recovering
recovering shards:
[1,2,24,26,29,33,35,37,40,41,42,46,48,51,54,66,76,95,100,101,122,123,127]
-----
I was going to try adding an OSD to my home cluster using one of the 4GB Raspberry Pis today, but it appears that the Ubuntu Bionic arm64 repo is missing a bunch of packages:
$ sudo grep ^Package: /var/lib/apt/lists/download.ceph.com_debian-nautilus_dists_bionic_main_binary-arm64_Packages
Package: ceph-deploy
Package: ceph-mgr-dashboard
Package: ceph-mgr-diskprediction-cloud
Package: ceph-mgr-diskprediction-local
Package: ceph-mgr-k8sevents
Package: ceph-mgr-rook
Package: ceph-mgr-ssh
Package: cephfs-shell
Package: libcephfs-java
Package: python-ceph-argparse
Package: python3-ceph-argparse
It looks like the packages were built, but for some reason the Packages file is missing them:
http://download.ceph.com/debian-nautilus/pool/main/c/ceph/
Could someone look into why the repo isn't finding these packages?
Thanks,
Bryan
Hi,
I have a weird situation where an OSD's rocksdb fails to compact, because
the OSD became full and the osd-full-ratio was 1.0 (not a good idea, I
know).
Hitting "bluefs enospc" while compacting:
-376> 2019-12-18 15:48:16.492 7f2e0a5ac700 1 bluefs _allocate failed to
allocate 0x40da486 on bdev 1, free 0x38b0000; fallback to bdev 2
-376> 2019-12-18 15:48:16.492 7f2e0a5ac700 -1 bluefs _allocate failed to
allocate 0x on bdev 2, dne
-376> 2019-12-18 15:48:16.492 7f2e0a5ac700 -1 bluefs _flush_range
allocated: 0x0 offset: 0x0 length: 0x40da486
-376> 2019-12-18 15:48:16.500 7f2e0a5ac700 -1
/build/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int
BlueFS::_flush_range(BlueFS::FileWriter*, uin
t64_t, uint64_t)' thread 7f2e0a5ac700 time 2019-12-18 15:48:16.499599
/build/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 ==
"bluefs enospc")
So my idea is to copy out rocksdb somewhere else (bluefs-export), compact,
then copy back in. Is there a way to do this? Mounting bluefs seems to be
part of the OSD code, so there's no easy way to do this it seems.
Because the OSD died at 100% full, I can't do bluefs-bdev-expand, and
repair/fsck fail too.
Thanks in advance.
Hello,
Anybody seeing the Prometheus endpoint hanging with the new 13.2.7 release?
With 13.2.6 the endpoint would respond with a payload of 15MB in less than
10 seconds.
Now, if you restart ceph-mgr, the Prometheus endpoint responds quickly for
the first run, then successive runs get slower and slower, until it takes
several minutes.
I have no customization for the mgr module. Except for the Prometheus
module, the "status" module and "Zabbix" module are working fine.
This is on Ubuntu 16 LTS:
ii ceph-mgr 13.2.7-1xenial
amd64 manager for the ceph distributed storage system
I'd love to know if there's a way to diagnose this issue - I tried upping
the debug ms level but that doesn't seem to yield useful information.
I don't know if this useful, but "prometheus self-test" is fine too.
$ sudo ceph tell mgr.0 prometheus self-test
Self-test OK
pchoi@pchoi-desktop:~$ ceph mgr module ls
{
"enabled_modules": [
"prometheus",
"status",
"zabbix"
],
"disabled_modules": [
{
"name": "balancer",
"can_run": true,
"error_string": ""
},
{
"name": "crash",
"can_run": true,
"error_string": ""
},
{
"name": "dashboard",
"can_run": true,
"error_string": ""
},
{
"name": "hello",
"can_run": true,
"error_string": ""
},
{
"name": "influx",
"can_run": false,
"error_string": "influxdb python module not found"
},
{
"name": "iostat",
"can_run": true,
"error_string": ""
},
{
"name": "localpool",
"can_run": true,
"error_string": ""
},
{
"name": "restful",
"can_run": true,
"error_string": ""
},
{
"name": "selftest",
"can_run": true,
"error_string": ""
},
{
"name": "smart",
"can_run": true,
"error_string": ""
},
{
"name": "telegraf",
"can_run": true,
"error_string": ""
},
{
"name": "telemetry",
"can_run": true,
"error_string": ""
}
]
}
pchoi@pchoi-desktop:~$ ceph mgr services
{
"prometheus": "http://woodenbox2:9283/"
}
I want run two realm on one ceph cluster, but I found rgw will use only one .rgw.root pool save rgw meta data, so , if run two realm on one ceph cluster, will be error ?
黄明友
IT基础架构部经理
V.Photos 云摄影
移动电话: +86 13540630430
客服电话：400 - 806 - 5775
电子邮件: hmy(a)v.photos
官方网址: www.v.photos
上海 黄浦区中山东二路88号外滩SOHO3Q F栋 2层
北京 朝阳区光华路9号光华路SOHO二期南二门SOHO3Q 1层
广州 天河区林和中路136号天誉花园二期3Wcoffice 天誉青创社区
深圳 南山区蛇口网谷科技大厦二期A座102网谷双创街 1层
成都 成华区建设路世贸广场 7层
Hi,
We have been rebalacing our cluster (CRUSH policy change for cephfs
metadata pool). The rebalancing is done ... but now there are PG
deep-scrubs going on (mostly on the pool that got almost completely
backfilled). We have had the no-(deep)-scrub flags active ... but that
does not prevent new (deep-)scrubs from happening. Also disabling them
completely on the specific pool did not help ... The oldest deep-scrub
was from the 14th of december ...
Is there logic coded somewhere that a PG that got backfilled needs to be
deep-scrubbed after that? I remember this was a thing on filestore (to
check integrity) ... should not be needed anymore with bluestore, right
(CRC checking)?
All this on Mimic 13.2.8
Thanks,
Stefan
P.s. I updated https://tracker.ceph.com/issues/11202
--
| BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
This is the fifth release of the Ceph Nautilus release series. Among the many
notable changes, this release fixes a critical BlueStore bug that was introduced
in 14.2.3. All Nautilus users are advised to upgrade to this release.
For the complete changelog entry, please visit the release blog at
https://ceph.io/releases/v14-2-5-nautilus-released/
Notable Changes
---------------
Critical fix:
* This release fixes a `critical BlueStore bug <https://tracker.ceph.com/issues/42223>`_
introduced in 14.2.3 (and also present in 14.2.4) that can lead to data
corruption when a separate "WAL" device is used.
New health warnings:
* Ceph will now issue health warnings if daemons have recently crashed. Ceph
has been collecting crash reports since the initial Nautilus release, but the
health alerts are new. To view new crashes (or all crashes, if you've just
upgraded)::
ceph crash ls-new
To acknowledge a particular crash (or all crashes) and silence the health warning::
ceph crash archive <crash-id>
ceph crash archive-all
* Ceph will now issue a health warning if a RADOS pool has a ``pg_num``
value that is not a power of two. This can be fixed by adjusting
the pool to a nearby power of two::
ceph osd pool set <pool-name> pg_num <new-pg-num>
Alternatively, the warning can be silenced with::
ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false
* Ceph will issue a health warning if a RADOS pool's ``size`` is set to 1
or, in other words, if the pool is configured with no redundancy. Ceph will
stop issuing the warning if the pool size is set to the minimum
recommended value::
ceph osd pool set <pool-name> size <num-replicas>
The warning can be silenced with::
ceph config set global mon_warn_on_pool_no_redundancy false
* A health warning is now generated if the average osd heartbeat ping
time exceeds a configurable threshold for any of the intervals
computed. The OSD computes 1 minute, 5 minute and 15 minute
intervals with average, minimum and maximum values. New configuration
option `mon_warn_on_slow_ping_ratio` specifies a percentage of
`osd_heartbeat_grace` to determine the threshold. A value of zero
disables the warning. New configuration option `mon_warn_on_slow_ping_time`
specified in milliseconds over-rides the computed value, causes a warning
when OSD heartbeat pings take longer than the specified amount.
A new admin command, `ceph daemon mgr.# dump_osd_network [threshold]`, will
list all connections with a ping time longer than the specified threshold or
value determined by the config options, for the average for any of the 3 intervals.
Another new admin command, `ceph daemon osd.# dump_osd_network [threshold]`,
will do the same but only including heartbeats initiated by the specified OSD.
Changes in the telemetry module:
* The telemetry module now has a 'device' channel, enabled by default, that
will report anonymized hard disk and SSD health metrics to telemetry.ceph.com
in order to build and improve device failure prediction algorithms. Because
the content of telemetry reports has changed, you will need to re-opt-in
with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show
ceph telemetry show device # specifically show the device channel
If you are not comfortable sharing device metrics, you can disable that
channel first before re-opting-in:
ceph config set mgr mgr/telemetry/channel_crash false
ceph telemetry on
* The telemetry module now reports more information about CephFS file systems,
including:
- how many MDS daemons (in total and per file system)
- which features are (or have been) enabled
- how many data pools
- approximate file system age (year + month of creation)
- how many files, bytes, and snapshots
- how much metadata is being cached
We have also added:
- which Ceph release the monitors are running
- whether msgr v1 or v2 addresses are used for the monitors
- whether IPv4 or IPv6 addresses are used for the monitors
- whether RADOS cache tiering is enabled (and which mode)
- whether pools are replicated or erasure coded, and
which erasure code profile plugin and parameters are in use
- how many hosts are in the cluster, and how many hosts have each type of daemon
- whether a separate OSD cluster network is being used
- how many RBD pools and images are in the cluster, and how many pools have RBD mirroring enabled
- how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use
- aggregate stats about the CRUSH map, like which algorithms are used, how
big buckets are, how many rules are defined, and what tunables are in
use
If you had telemetry enabled, you will need to re-opt-in with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show # see everything
ceph telemetry show basic # basic cluster info (including all of the new info)
OSD:
* A new OSD daemon command, 'dump_recovery_reservations', reveals the
recovery locks held (in_progress) and waiting in priority queues.
* Another new OSD daemon command, 'dump_scrub_reservations', reveals the
scrub reservations that are held for local (primary) and remote (replica) PGs.
RGW:
* RGW now supports S3 Object Lock set of APIs allowing for a WORM model for
storing objects. 6 new APIs have been added put/get bucket object lock,
put/get object retention, put/get object legal hold.
* RGW now supports List Objects V2
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-14.2.5.tar.gz
* For packages, see http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: ad5bd132e1492173c85fda2cc863152730b16a92
--
Abhishek Lekshmanan
SUSE Software Solutions Germany GmbH
