Hi all!
In the context of a repository-type project, we need to address a
situation where we cannot use periodic checks in Ceph (scrubbing) due to
the project's nature. Instead, we need the ability to write a checksum
into the metadata of the uploaded file via API. In this context, we are
not concerned about individual file parts, but rather the file as a
whole. Users will calculate the checksum and write it. Based on this
hash, we should be able to trigger a check of the given files. We are
aware that tools like s3cmd can write MD5 hashes to file metadata, but
is there a more general approach? Does anyone have experience with this,
or can you suggest a tool that can accomplish this?
Thx
Michal
Hi everyone,
I confirmed that write performance has increased too much even if I apply
just SSD for the index pool of rgw.
I know that ~200 Bytes per object in the index pool is created.
When I checked the index pool size, it's around 300 bytes ~ 400 bytes
calculated.
like me, If it uses index pool applying separated SSD devices, I guess
there is the tuning factor like block size to reduce index pool size.
Is there the tuning factor or recommendation about separated SSD for the
index pool?
Hi everyone,
I saw the bluestore can separate block.db, block.wal.
In my case, I'd like to apply hybrid device which uses SSD, HDD to improve
the small data write performance.
but I don't have enough SSD to cover block.db and block.wal.
so I think it can impact performance even though SSD applies into just
block.wal.
I just know that block.wal depends on rocksdb cache size as parameters. SSD
might not need too much.
1.
When I use SSD just into block.wal,
Does it impact the write performance of the small data?
2.
Should I make lv of ssd per osd for block.wal?
3.
How much SSD do I need for block.wal relative to HDD(if I have 100TB)?
I just jacked in a completely new, clean server and I've been trying to
get a Ceph (Pacific) monitor running on it.
The "ceph orch daemon add" appears to install all/most of what's
necessary, but when the monitor starts, it shuts down immediately, and
in the manner of Ceph containers immediately erases itself and the
container log, so it's not possible to see what its problem is.
I looked at manual installation, but the docs appear to be oriented
towards old-style non-container implementation and don't account for
the newer /var/lib/ceph/*fsid*/ approach.
Any tips?
Last few lines in the system journal are like this:
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.938+0000
7f26810ae700 4 rocksdb: (Original Log Time 2024/02/06-16:09:58.938432)
[compaction/compaction_job.cc:760] [default] compacted to: base level 6
level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 2]
max score 0.00, MB/sec: 351.7 rd, 351.7 wr, level 6, files in(4, 0)
out(2) MB in(92.8, 0.0) out(92.8), read-write-amplify(2.0) write-
amplify(1.0) OK, records in: 2858, records dropped: 0
output_compression: NoCompression
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]:
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.938+0000
7f26810ae700 4 rocksdb: (Original Log Time 2024/02/06-16:09:58.938452)
EVENT_LOG_v1 {"time_micros": 1707235798938446, "job": 6, "event":
"compaction_finished", "compaction_time_micros": 276718,
"compaction_time_cpu_micros": 73663, "output_level": 6,
"num_output_files": 2, "total_output_size": 97309398,
"num_input_records": 2858, "num_output_records": 2858,
"num_subcompactions": 1, "output_compression": "NoCompression",
"num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0,
"lsm_state": [0, 0, 0, 0, 0, 0, 2]}
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.940+0000
7f26810ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798941291,
"job": 6, "event": "table_file_deletion", "file_number": 14}
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.943+0000
7f26810ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798943980,
"job": 6, "event": "table_file_deletion", "file_number": 12}
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.946+0000
7f26810ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798946734,
"job": 6, "event": "table_file_deletion", "file_number": 10}
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.946+0000
7f26810ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798946789,
"job": 6, "event": "table_file_deletion", "file_number": 4}
Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.450+0000
7f26818af700 -1 received signal: Terminated from Kernel ( Could be
generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.450+0000
7f26818af700 -1 mon.dell02@-1(synchronizing) e161 *** Got Signal
Terminated ***
Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.450+0000
7f26818af700 1 mon.dell02@-1(synchronizing) e161 shutdown
Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.452+0000
7f2691a95880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling
all background work
Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.452+0000
7f2691a95880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete
Feb 06 11:09:59 dell02.mousetech.com bash[1357898]: ceph-278fcd86-0861-
11ee-a7df-9c5c8e86cf8f-mon-dell02
Hello,
I have an issue at my company where we have an underperforming Ceph
instance.
The issue that we have is that sometimes writing files to Ceph via S3 API
(our only option) takes up to 40s, which is too long for us.
We are a bit limited on what we can do to investigate why it's performing
so badly, because we have a service provider in between, so getting to the
bottom of this really is not that easy.
That being said, the way we use the S3 APi (again, Ceph under the hood) is
by writing all files (multiple millions) to the root, so we don't use *no*
folder-like structure e.g. we write */<uuid>* instead of */this/that/<uuid>*
.
The question is:
Does anybody know whether Ceph has performance gains when you create a
folder structure vs when you don't?
Looking at Ceph's documentation I could not find such information.
Best regards,
*Renann Prado*
hello, i was tried to create osd but when i run ceph command the output is like this :
root@pod-deyyaa-ceph1:~# sudo ceph -s
2024-02-02T16:01:23.627+0700 7fc762f37640 0 monclient(hunting): authenticate timed out after 300
[errno 110] RADOS timed out (error connecting to the cluster)
can anyone give me a hint or help me to solve this
kicked off
ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.1
and just MDS left to do but upgrade has been sitting for hours on this
root@rdx-00:~# ceph orch upgrade status
{
"target_image": "
quay.io/ceph/ceph@sha256:a4e86c750cc11a8c93453ef5682acfa543e3ca08410efefa30f520b54f41831f
",
"in_progress": true,
"which": "Upgrading all daemon types on all hosts",
"services_complete": [
"osd",
"mgr",
"mon",
"crash"
],
"progress": "1267/1306 daemons upgraded",
"message": "Currently upgrading osd daemons",
"is_paused": false
}
root@rdx-00:~#
message seems erroneous as all the OSDs seem to be done.
Can rbd image snapshotting be scheduled like CephFS snapshots? Maybe I missed it in the documentation but it looked like scheduling snapshots wasn’t a feature for block images. I’m still running Pacific. We’re trying to devise a sufficient backup plan for Cloudstack and other things residing in Ceph.
Thanks.
-jeremy
Hi
I have recently onboarded new OSDs into my Ceph Cluster. Previously, I had
44 OSDs of 1.7TiB each and was using it for about a year. About 1 year ago,
we onboarded an additional 20 OSDs of 14TiB each.
However I observed that many of the data were still being written onto the
original 1.7TiB OSDs instead of the 14TiB ones. Overtime, this caused a
bottleneck as the 1.7TiB OSDs reached nearfull capacity.
I have tried to perform a reweight (both crush reweight and reweight) to
reduce the number of PGs on each 1.7TiB. This worked temporarily but
resulted in many objects being misplaced and PGs being in a Warning state.
Subsequently I have also tried using crush-compat balancer mode instead of
upmap but did not see significant improvement. The latest changes I made
was to change backfill-threshold to 0.85, hoping that PGs will no longer be
assigned to OSDs that are >85% utilization. However, this did not change
the situation much as I see many OSDs above >85% utilization today.
Attached is a report from ceph report command. For now I have OUT-ed two of
my OSDs which have reached 95% capacity. I would greatly appreciate it if
someone can provide advice on this matter.
Thanks
Jasper Tan
--
--
--
*The contents of this e-mail message and any attachments are
confidential and are intended solely
for addressee. The information may
also be legally privileged. This transmission is sent in trust, for
the
sole purpose of delivery to the intended recipient. If you have received
this transmission in error,
any use, reproduction or dissemination of this
transmission is strictly prohibited. If you are not the
intended recipient,
please immediately NOTIFY the sender by reply e-mail or phone and DELETE
this message and its attachments, if any.*
Back when I was battline Octopus, I had problems getting ganesha's NFS
to work reliably. I resolved this by doing a direct (ceph) mount on my
desktop machine instead of an NFS mount.
I've since been plagued by ceph "laggy OSD" complaints that appear to
be due to a non-responsive client and I'm suspecting that the client in
question is the desktop machine when it's suspended while the ceph
mount is in effect.
So the question is: Should ceph native mounts be used on general client
machines which may hibernate or otherwise go offline?
