Hi,
We've done our fair share of Ceph cluster upgrades since Hammer, and
have not seen much problems with them. I'm now at the point that I have
to upgrade a rather large cluster running Luminous and I would like to
hear from other users if they have experiences with issues I can expect
so that I can anticipate on them beforehand.
As said, the cluster is running Luminous (12.2.13) and has the following
services active:
services:
mon: 3 daemons, quorum osdnode01,osdnode02,osdnode04
mgr: osdnode01(active), standbys: osdnode02, osdnode03
mds: pmrb-3/3/3 up {0=osdnode06=up:active,1=osdnode08=up:active,2=osdnode07=up:active}, 1 up:standby
osd: 116 osds: 116 up, 116 in;
rgw: 3 daemons active
Of the OSD's, we have 11 SSD's and 105 HDD. The capacity of the cluster
is 1.01PiB.
We have 2 active crush-rules on 18 pools. All pools have a size of 3 there is a total of 5760 pgs.
{
"rule_id": 1,
"rule_name": "hdd-data",
"ruleset": 1,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -10,
"item_name": "default~hdd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 2,
"rule_name": "ssd-data",
"ruleset": 2,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -21,
"item_name": "default~ssd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
rbd -> crush_rule: hdd-data
.rgw.root -> crush_rule: hdd-data
default.rgw.control -> crush_rule: hdd-data
default.rgw.data.root -> crush_rule: ssd-data
default.rgw.gc -> crush_rule: ssd-data
default.rgw.log -> crush_rule: ssd-data
default.rgw.users.uid -> crush_rule: hdd-data
default.rgw.usage -> crush_rule: ssd-data
default.rgw.users.email -> crush_rule: hdd-data
default.rgw.users.keys -> crush_rule: hdd-data
default.rgw.meta -> crush_rule: hdd-data
default.rgw.buckets.index -> crush_rule: ssd-data
default.rgw.buckets.data -> crush_rule: hdd-data
default.rgw.users.swift -> crush_rule: hdd-data
default.rgw.buckets.non-ec -> crush_rule: ssd-data
DB0475 -> crush_rule: hdd-data
cephfs_pmrb_data -> crush_rule: hdd-data
cephfs_pmrb_metadata -> crush_rule: ssd-data
All but four clients are running Luminous, the four are running Jewel
(that needs upgrading before proceeding with this upgrade).
So, normally, I would 'just' upgrade all Ceph packages on the
monitor-nodes and restart mons and then mgrs.
After that, I would upgrade all Ceph packages on the OSD nodes and
restart all the OSD's. Then, after that, the MDSes and RGWs. Restarting
the OSD's will probably take a while.
If anyone has a hint on what I should expect to cause some extra load or
waiting time, that would be great.
Obviously, we have read
https://ceph.com/releases/v14-2-0-nautilus-released/ , but I'm looking
for real world experiences.
Thanks!
--
Mark Schouten | Tuxis B.V.
KvK: 74698818 | http://www.tuxis.nl/
T: +31 318 200208 | info(a)tuxis.nl
Hi,
Is there anybody know about list-type=2 request?
GET /bucket?list-type=2&max-keys=2
We faced yesterday the 2nd big objectstore cluster outage due to this request. 1 user made the cluster down totally. The normal ceph iostat read operation is below 30k, when they deployed their release it jumped up to 350k which made rados-gateway died under the haproxy.
Why ceph is so sensitive for this or what is this actually? I don't even found anything in google.
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Hi,
just wanted to ask if it is intentional that
http://ceph.com/pgcalc/
results in a 404 error?
is there any alternative url?
it is still linked from the offical docs.
with kind regards
Dominik
Is this happening to anyone else? After this command:
ceph health mute AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED 2w
The 'dashboard' shows 'Health OK', then after a few hours (perhaps a
mon leadership change), it's back to 'degraded' and
'AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED: mons are allowing insecure
global_id reclaim'
Pacific, 16.2.4 all in docker containers.
Hi,
Want to have logs from cluster on Graylog but seems like CEPH send empty
"host" field. Any one can help ?
CEPH 16.2.3
# ceph config dump | grep graylog
global advanced clog_to_graylog true
global advanced clog_to_graylog_host xx.xx.xx.xx
global basic err_to_graylog true
global basic log_graylog_host xx.xx.xx.xx *
global basic log_to_graylog true
I see that my Graylog is hit by traffic from ceph on port 12201 udp and
parsed by GELF udp
Grylog logs:
2021-07-01 12:16:57,355 ERROR:
org.graylog2.shared.buffers.processors.DecodingProcessor - Error
processing message RawMessage{id=3ad5c6a1-da66-11eb-a55c-0242ac120005,
messageQueueId=2810784, codec=gelf, payloadSize=340,
timestamp=2021-07-01T12:16:57.354Z, remoteAddress=/xx.xx.xx.xx:34049}
java.lang.IllegalArgumentException: GELF message
<3ad5c6a1-da66-11eb-a55c-0242ac120005> (received from
<xx.xx.xx.xx:34049>) has empty mandatory "host" field.
at
org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:247)
~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140)
~[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153)
~[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94)
[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90)
[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47)
[graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143)
[graylog.jar:?]
at
com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66)
[graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Best regards Milosz
I'm still attempting to build a ceph cluster and I'm currently getting
nowhere very very quickly. From what I can tell I have a slightly unstable
setup and I'm yet to work out why.
I currently have 24 servers and I'm planning to increase this to around 48
These servers are in three groups with different types of disks (and
number) in each type.
Currently I'm having an issue where every time I add a new server it adds
the osd on the node and then a few random ods on the current hosts will all
fall over and I'll only be able to get them up again by restart the daemons.
I'm using cephadm. and the network is a QDR based IB network running IP
over IB so its meant to be 40G but currently is behaving more like 10G
(when I've tested it) Its still faster than the 1G management network I've
also got.
The machines are mostly running debian. There are a few machine running
CentOS7 I'm meaning to redeploy when I get the time (so I can upgrade to
Pacific)
I'm running Octopus 15.2.13, I'm more than happy to change stuff I'm still
trying to learn stuff so there is no data that I care about quite yet, I
was looking for more stability before I go there.
I really just want to know where to look for the problems rather than any
exact answers, I'm yet to see any clues that might help
Thanks in advance
Peter Childs
Hello,
After having done a rolling reboot of my Octopus 15.2.13 cluster of 8 nodes cephadm does not find python3 on the node and hence I get quite a few of the following warnings:
[WRN] CEPHADM_HOST_CHECK_FAILED: 7 hosts fail cephadm check
host ceph1f failed check: Can't communicate with remote host `ceph1f`, possibly because python3 is not installed there: [Errno 32] Broken pipe
Here is the full stack trace from cephadm:
2021-07-06T06:03:20.798410+0000 mgr.ceph1a.xxqpph [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec(name=all-available-devices->placement=PlacementSpec(host_pattern='*'), service_id='all-available-devices', service_type='osd', data_devices=DeviceSelection(all=True), osd_id_claims={}, unmanaged=False, filter_logic='AND', preview_only=False): Can't communicate with remote host `ceph1d`, possibly because python3 is not installed there: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/module.py", line 1015, in _remote_connection
conn, connr = self._get_connection(addr)
File "/usr/share/ceph/mgr/cephadm/module.py", line 978, in _get_connection
sudo=True if self.ssh_user != 'root' else False)
File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 34, in __init__
self.gateway = self._make_gateway(hostname)
File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 44, in _make_gateway
self._make_connection_string(hostname)
File "/lib/python3.6/site-packages/execnet/multi.py", line 134, in makegateway
gw = gateway_bootstrap.bootstrap(io, spec)
File "/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 102, in bootstrap
bootstrap_exec(io, spec)
File "/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 46, in bootstrap_exec
"serve(io, id='%s-slave')" % spec.id,
File "/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 78, in sendexec
io.write((repr(source) + "\n").encode("ascii"))
File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 409, in write
self._write(data)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/module.py", line 1019, in _remote_connection
raise execnet.gateway_bootstrap.HostNotFound(msg)
execnet.gateway_bootstrap.HostNotFound: Can't communicate with remote host `ceph1d`, possibly because python3 is not installed there: [Errno 32] Broken pipe
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in _apply_all_services
if self._apply_service(spec):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 450, in _apply_service
self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec))
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 51, in create_from_spec
ret = create_from_spec_one(self.prepare_drivegroup(drive_group))
File "/usr/share/ceph/mgr/cephadm/utils.py", line 65, in forall_hosts_wrapper
return CephadmOrchestrator.instance._worker_pool.map(do_work, vals)
File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/usr/share/ceph/mgr/cephadm/utils.py", line 59, in do_work
return f(*arg)
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 47, in create_from_spec_one
host, cmd, replace_osd_ids=osd_id_claims.get(host, []), env_vars=env_vars
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 56, in create_single_host
out, err, code = self._run_ceph_volume_command(host, cmd, env_vars=env_vars)
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 271, in _run_ceph_volume_command
error_ok=True)
File "/usr/share/ceph/mgr/cephadm/module.py", line 1100, in _run_cephadm
with self._remote_connection(host, addr) as tpl:
File "/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/usr/share/ceph/mgr/cephadm/module.py", line 1046, in _remote_connection
raise OrchestratorError(msg) from e
orchestrator._interface.OrchestratorError: Can't communicate with remote host `ceph1d`, possibly because python3 is not installed there: [Errno 32] Broken pipe
I checked directly on the nodes and I can execute "python3" command and I can also SSH into all nodes with the following test command:
ssh -F ssh_config -i ~/cephadm_private_key root@nodeX
So I don't really understand what could have broken the cephadm orchestrator... Any ideas? The cephfs itself is still working.
Best regards,
Mabi