Hi Folks,
Perf meeting is on in ~35 minutes! Let's continue the conversation from
last week about nvdimms. Today we have new test results showing the
effect of PGLog omap removal and pglog removal (mostly). Feel free to
add your own topic as well!
Etherpad:
https://pad.ceph.com/p/performance_weekly
Bluejeans:
https://bluejeans.com/908675367
Thanks,
Mark
ceph-daemon and mgr/ssh are now (or soon to be) cephadm:
https://github.com/ceph/ceph/pull/32193
We want to merge this as quickly as possible to unblock other
conflicting work.
The key change is that ceph-mgr-ssh is now ceph-mgr-cephadm. Note that
this package is only needed on master (not nautilus or earlier). And
ceph-mgr-ssh isn't needed on nautilus or earlier (tho it was there
before). Here's my attempt:
https://github.com/ceph/ceph-container/pull/1533
Will that work? Or can you adjust as needed?
A bunch of the qa tests mostly on the latest-master-devel image, but
that's won't work until the above two PRs merge. So basically as soon as
we can merge something into ceph-container that includes the new package
(and neither for nautilus), we can merge the ceph PR too.
Thanks!
sage
Hi everyone.
The next DocuBetter meeting is scheduled for:
11 Dec 2019 0930 PST
11 Dec 2019 1730 UTC
12 Dec 2019 0330 AEST
Etherpad: https://pad.ceph.com/p/Ceph_Documentation
Meeting: https://bluejeans.com/908675367
Agenda: 1. Resolution of outstanding Sphinx issues
2. Documentation Bug Progress Report
3. Old Business -- CephFS manual
4. Old Business -- RADOS Documentation
5. Old Business -- docs bug reporting and improvement-request
procedure
Thanks, everyone.
Zac Dover
The bucket index logs used for multisite replication are currently
stored in omap on the bucket index shards, along with the rest of the
bucket index entries. Storing them in the index was a natural choice,
because cls_rgw can write these log entries atomically when completing a
bucket index transaction.
To replicate a bucket, other zones process each of its bucket index
shard logs independently, and store sync status markers with their
position in each shard. This tight coupling between the replication
strategy and the bucket's sharding scheme is the main challenge to
supporting bucket resharding in multisite, because shuffling these log
entries to a new set of shards would invalidate the sync status markers
stored in other zones.
My proposal, then, is to move the replication logs out of bucket index
shards into a single log per bucket, and extend the consistency model to
make up for the lack of atomic writes that we get from cls_rgw.
The existing consistency model for object writes involves a) calling
cls_rgw to prepare a bucket index transaction, b) writing the object's
head to the data pool, then c) calling cls_rgw to complete the
transaction. Since the write in b) is what makes the object visible to
GET requests, we can reply to the client without waiting for c) to
finish. If either b) or c) fails, the next bucket listing will find an
entry that was prepared but not completed, and we'll check whether the
head object exists and use the 'dir suggest' call to update the bucket
index accordingly.
If we move the replication log to a separate object, we'll need to write
to that as well before completing the transaction. And when dir suggest
finds head objects for uncompleted transactions, it can (re)write their
replication log entries before updating the bucket index. This recovery
means that we can still reply to the client before writing to the
replication log, so the client won't see any extra latency.
This change also gives us the opportunity to move away from omap and the
challenges associated with trimming. Yehuda wrote cls_fifo in
https://github.com/ceph/ceph/pull/30797 with the datalog in mind, and
that could be a good fit for these bucket replication logs as well.
Hi everyone,
The next Cephalocon is coming up on March 3-5 in Seoul! The CFP is open
until Friday (get your talks in!). We expect to have the program
ready for the first week of January. Registration (early bird) will be
available soon.
We're also looking for sponsors for the conference. The prospectus is
available here:
https://ceph.io/wp-content/uploads/2019/12/sponsor-Cephalocon20-112719.pdf
Thanks!
sage
Hi Ken
There is a lot of interest in getting the Python3/CentOS8 pr in master
merged [0], but we can't do that at the moment since the dependencies
are missing.
As you know, we've been building everything we need in the ceph-el8
copr repo [1], and I would like your take on merging the PR in master
while we wait to get packages in EPEL8. This would unblock development
in master that can't move forward unless the CentOS8 support is there.
I value your guidance here, and I'd like to know if you are OK with
this approach, or if you have any other ideas to achieve progress.
Thanks!
[0] https://github.com/ceph/ceph/pull/25969
[1] https://copr.fedorainfracloud.org/coprs/ktdreyer/ceph-el8/
During CDM on Wednesday I suggested that ceph-daemon should probably be
renamed before octopus (and before the name sticks). In speech "ceph
daemon" is confusing, and the name doesn't really reflect what it
is:
a user-facing tool
- to bootstrap a new cluster,
- launch a (containerized) shell,
- enter an existing daemon container,
- tail a daemon's log, or
- adopt a daemon deployed with a legacy tool (ceph-deploy, ceph-ansible,
etc) into a ceph-daemon style container
- remove all trace of a cluster from the localhost,
and an internal tool used by ssh-orch to
- deploy or remove a container running a ceph daemon
- start, stop, or update an existing container
- run ceph-volume (to gather device inventory, create osds, zap, etc.)
- gather a host inventory of services (containers)
The original tool was more like "ceph daemon tool" but it was shortened to
ceph-daemon at the start.
We voted on some alternatives here:
https://pad.ceph.com/p/ceph-daemon-tool
but the winner is currently 'cephctl' with is IMO a non-starteer.
Usually 'ctl' utilities are for interacting with running systems/daemons
via some runtime API--that's basically what the 'ceph' CLI utility is.
ceph-daemon is more akin to kubeadm or ceph-deploy or something like
that.
So... I shortened/pruned the list again. Please weigh in. And if you
have a bright idea for a better name, feel free to add it.
Thanks!
sage
Hi, @Yehuda Sadeh-Weinraub <yehuda(a)redhat.com> @Casey Bodley
<cbodley(a)redhat.com> @Matt Benjamin <mbenjamin(a)redhat.com> and Cephers
We met a problem with the Elastic Search sync module: bucket with custom
placement could not be synced to the target zone. The target zone tries to
create a bucket instance based on the placement name, but the target zone
does have the placement. logs as following:
meta sync: ERROR: can't store key: bucket.instance: *
data sync: ERROR: failed to fetch bucket instance info for *
ERROR: select_bucket_placement() returned -22
we are going to fix this issue. here are some questions:
1. should we sync those buckets to the ES zone? it seems not necessary
2. should we sync placements? if we sync placements to the target zone, we
also need to create rados pools in the target zone
thanks
Chang Liu
