24 Nov
2023
24 Nov
'23
4:48 p.m.
Hi,
After updating from 17.2.6 to 17.2.7 with cephadm, our cluster went into
MDS_DAMAGE state. We had some prior issues with faulty kernel clients
not releasing capabilities, therefore the update might just be a
coincidence.
`ceph tell mds.cephfs:0 damage ls` lists 56 affected files all with
these general details:
{
"damage_type": "dentry",
"id": 123456,
"ino": 1234567890,
"frag": "*",
"dname": "some-filename.ext",
"snap_id": "head",
"path": "/full/path/to/file"
}
The behaviour upon trying to access file information in the (Kernel
mounted) filesystem is a bit inconsistent. Generally, the first `stat`
call seems to result in "Input/output error", the next call provides all
`stat` data as expected from an undamaged file. The file can be read
with `cat` with full and correct content (verified with backup) once the
stat call succeeds.
Scrubbing the affected subdirectories with `ceph tell mds.cephfs:0 scrub
start /path/to/dir/ recursive,repair,force` does not fix the issue.
Trying to delete the file results in an "Input/output error". If the
stat calls beforehand succeeded, this also crashes the active MDS with
these messages in the system journal:
Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]:
mds.0.cache.den(0x10012271195 DisplaySettings.json) newly corrupt dentry to be committed:
[dentry
#0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
[1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 state=1073741824 |
inodepin=1 0x56413e1e2780]
Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: log_channel(cluster) log [ERR] :
MDS abort because newly corrupt dentry to be committed: [dentry
#0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
[1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 state=1073741824 |
inodepin=1 0x56413e1e2780]
Nov 24 14:21:15 iceph-18.servernet
ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]:
2023-11-24T13:21:15.654+0000 7f3fdcde0700 -1 mds.0.cache.den(0x10012271195
DisplaySettings.json) newly corrupt dentry to be committed: [dentry
#0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
[1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x1001>
Nov 24 14:21:15 iceph-18.servernet
ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]:
2023-11-24T13:21:15.654+0000 7f3fdcde0700 -1 log_channel(cluster) log [ERR] : MDS abort
because newly corrupt dentry to be committed: [dentry
#0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
[1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012>
Nov 24 14:21:15 iceph-18.servernet
ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc:
In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 time
2023-11-24T13:21:15.655088+0000
Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc:
In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 time
2023-11-24T13:21:15.655088+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc:
937: ceph_abort_msg("abort() called")
ceph version 17.2.7
(b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)
1: (ceph::__ceph_abort(char
const*, int, char const*, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)+0xd7)
[0x7f3fe5a1cb03]
2:
(MDSRank::abort(std::basic_string_view<char, std::char_traits<char> >)+0x7d)
[0x5640f2e6fa2d]
3:
(CDentry::check_corruption(bool)+0x740) [0x5640f30e4820]
4:
(EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned
char)+0x47) [0x5640f2f41877]
5:
(EOpen::add_clean_inode(CInode*)+0x121) [0x5640f2f49fc1]
6:
(Locker::adjust_cap_wanted(Capability*, int, int)+0x426) [0x5640f305e036]
7:
(Locker::process_request_cap_release(boost::intrusive_ptr<MDRequestImpl>&,
client_t, ceph_mds_request_release const&, std::basic_string_view<char,
std::char_traits<char> >)+0x599) [0x5640f307f7e9]
8:
(Server::handle_client_request(boost::intrusive_ptr<MClientRequest const>
const&)+0xc06) [0x5640f2f2a7c6]
9:
(Server::dispatch(boost::intrusive_ptr<Message const> const&)+0x13c)
[0x5640f2f2ef6c]
10:
(MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5db)
[0x5640f2e7727b]
11:
(MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const>
const&)+0x5c) [0x5640f2e778bc]
12:
(MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1bf)
[0x5640f2e60c2f]
13:
(Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478)
[0x7f3fe5c97ed8]
14: (DispatchQueue::entry()+0x50f)
[0x7f3fe5c9531f]
15:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe5d5f381]
16:
/lib64/libpthread.so.0(+0x81ca) [0x7f3fe4a0b1ca]
17: clone()
Deleting the file with cephfs-shell also does give Input/output error (5).
Does anyone have an idea on how to proceed here? I am perfectly fine
with loosing the affected files, they can all be easily restored from
backup.
Cheers
Sebastian
24 Nov
24 Nov
8:30 p.m.
Hi Sebastian,
You can find some more discussion and fixes for this type of fs
corruption here:
https://www.spinics.net/lists/ceph-users/msg76952.html
--
Dan van der Ster
CTO
Clyso GmbH
p: +49 89 215252722 | a: Vancouver, Canada
w: https://clyso.com | e: dan.vanderster(a)clyso.com
We are hiring: https://www.clyso.com/jobs/
On Fri, Nov 24, 2023 at 5:48 AM Sebastian Knust
<sknust(a)physik.uni-bielefeld.de> wrote:
>
> Hi,
>
> After updating from 17.2.6 to 17.2.7 with cephadm, our cluster went into
> MDS_DAMAGE state. We had some prior issues with faulty kernel clients
> not releasing capabilities, therefore the update might just be a
> coincidence.
>
> `ceph tell mds.cephfs:0 damage ls` lists 56 affected files all with
> these general details:
>
> {
> "damage_type": "dentry",
> "id": 123456,
> "ino": 1234567890,
> "frag": "*",
> "dname": "some-filename.ext",
> "snap_id": "head",
> "path": "/full/path/to/file"
> }
>
> The behaviour upon trying to access file information in the (Kernel
> mounted) filesystem is a bit inconsistent. Generally, the first `stat`
> call seems to result in "Input/output error", the next call provides all
> `stat` data as expected from an undamaged file. The file can be read
> with `cat` with full and correct content (verified with backup) once the
> stat call succeeds.
>
> Scrubbing the affected subdirectories with `ceph tell mds.cephfs:0 scrub
> start /path/to/dir/ recursive,repair,force` does not fix the issue.
>
> Trying to delete the file results in an "Input/output error". If the
> stat calls beforehand succeeded, this also crashes the active MDS with
> these messages in the system journal:
> > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]:
mds.0.cache.den(0x10012271195 DisplaySettings.json) newly corrupt dentry to be committed:
[dentry
#0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
[1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 state=1073741824 |
inodepin=1 0x56413e1e2780]
> > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: log_channel(cluster) log
[ERR] : MDS abort because newly corrupt dentry to be committed: [dentry
#0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
[1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 state=1073741824 |
inodepin=1 0x56413e1e2780]
> > Nov 24 14:21:15 iceph-18.servernet
ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]:
2023-11-24T13:21:15.654+0000 7f3fdcde0700 -1 mds.0.cache.den(0x10012271195
DisplaySettings.json) newly corrupt dentry to be committed: [dentry
#0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
[1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x1001>
> > Nov 24 14:21:15 iceph-18.servernet
ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]:
2023-11-24T13:21:15.654+0000 7f3fdcde0700 -1 log_channel(cluster) log [ERR] : MDS abort
because newly corrupt dentry to be committed: [dentry
#0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
[1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012>
> > Nov 24 14:21:15 iceph-18.servernet
ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc:
In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 time
2023-11-24T13:21:15.655088+0000
> > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc:
In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 time
2023-11-24T13:21:15.655088+0000
> >
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc:
937: ceph_abort_msg("abort() called")
> >
> > ceph version 17.2.7
(b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)
> > 1:
(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)+0xd7)
[0x7f3fe5a1cb03]
> > 2:
(MDSRank::abort(std::basic_string_view<char, std::char_traits<char> >)+0x7d)
[0x5640f2e6fa2d]
> > 3:
(CDentry::check_corruption(bool)+0x740) [0x5640f30e4820]
> > 4:
(EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned
char)+0x47) [0x5640f2f41877]
> > 5:
(EOpen::add_clean_inode(CInode*)+0x121) [0x5640f2f49fc1]
> > 6:
(Locker::adjust_cap_wanted(Capability*, int, int)+0x426) [0x5640f305e036]
> > 7:
(Locker::process_request_cap_release(boost::intrusive_ptr<MDRequestImpl>&,
client_t, ceph_mds_request_release const&, std::basic_string_view<char,
std::char_traits<char> >)+0x599) [0x5640f307f7e9]
> > 8:
(Server::handle_client_request(boost::intrusive_ptr<MClientRequest const>
const&)+0xc06) [0x5640f2f2a7c6]
> > 9:
(Server::dispatch(boost::intrusive_ptr<Message const> const&)+0x13c)
[0x5640f2f2ef6c]
> > 10:
(MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5db)
[0x5640f2e7727b]
> > 11:
(MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const>
const&)+0x5c) [0x5640f2e778bc]
> > 12:
(MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1bf)
[0x5640f2e60c2f]
> > 13:
(Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478)
[0x7f3fe5c97ed8]
> > 14:
(DispatchQueue::entry()+0x50f) [0x7f3fe5c9531f]
> > 15:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe5d5f381]
> > 16:
/lib64/libpthread.so.0(+0x81ca) [0x7f3fe4a0b1ca]
> > 17: clone()
>
> Deleting the file with cephfs-shell also does give Input/output error (5).
>
> Does anyone have an idea on how to proceed here? I am perfectly fine
> with loosing the affected files, they can all be easily restored from
> backup.
>
> Cheers
> Sebastian
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
27 Nov
27 Nov
9:05 p.m.
Hello Sebastian,
On Fri, Nov 24, 2023 at 8:49 AM Sebastian Knust
<sknust(a)physik.uni-bielefeld.de> wrote:
Hi,
After updating from 17.2.6 to 17.2.7 with cephadm, our cluster went into
MDS_DAMAGE state. We had some prior issues with faulty kernel clients
not releasing capabilities, therefore the update might just be a
coincidence.
`ceph tell mds.cephfs:0 damage ls` lists 56 affected files all with
these general details:
{
"damage_type": "dentry",
"id": 123456,
"ino": 1234567890,
"frag": "*",
"dname": "some-filename.ext",
"snap_id": "head",
"path": "/full/path/to/file"
}
The behaviour upon trying to access file information in the (Kernel
mounted) filesystem is a bit inconsistent. Generally, the first `stat`
call seems to result in "Input/output error", the next call provides all
`stat` data as expected from an undamaged file. The file can be read
with `cat` with full and correct content (verified with backup) once the
stat call succeeds.
Scrubbing the affected subdirectories with `ceph tell mds.cephfs:0 scrub
start /path/to/dir/ recursive,repair,force` does not fix the issue.
Trying to delete the file results in an "Input/output error". If the
stat calls beforehand succeeded, this also crashes the active MDS with
these messages in the system journal:
I would **really** love to see the debug logs from the MDS. Please
upload them using ceph-post-file [1]. If you can reliably reproduce,
turn on more debugging:
Nov 24 14:21:15 iceph-18.servernet
ceph-mds[1946861]: mds.0.cache.den(0x10012271195 DisplaySettings.json) newly corrupt
dentry to be committed: [dentry
#0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
[1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 state=1073741824 |
inodepin=1 0x56413e1e2780]
Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: log_channel(cluster) log [ERR] :
MDS abort because newly corrupt dentry to be committed: [dentry
#0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
[1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 state=1073741824 |
inodepin=1 0x56413e1e2780]
Nov 24 14:21:15 iceph-18.servernet
ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]:
2023-11-24T13:21:15.654+0000 7f3fdcde0700 -1 mds.0.cache.den(0x10012271195
DisplaySettings.json) newly corrupt dentry to be committed: [dentry
#0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
[1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x1001>
Nov 24 14:21:15 iceph-18.servernet
ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]:
2023-11-24T13:21:15.654+0000 7f3fdcde0700 -1 log_channel(cluster) log [ERR] : MDS abort
because newly corrupt dentry to be committed: [dentry
#0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
[1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012>
Nov 24 14:21:15 iceph-18.servernet
ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc:
In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 time
2023-11-24T13:21:15.655088+0000
Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc:
In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 time
2023-11-24T13:21:15.655088+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc:
937: ceph_abort_msg("abort() called")
ceph version 17.2.7
(b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)
1: (ceph::__ceph_abort(char
const*, int, char const*, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)+0xd7)
[0x7f3fe5a1cb03]
2:
(MDSRank::abort(std::basic_string_view<char, std::char_traits<char> >)+0x7d)
[0x5640f2e6fa2d]
3:
(CDentry::check_corruption(bool)+0x740) [0x5640f30e4820]
4:
(EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned
char)+0x47) [0x5640f2f41877]
5:
(EOpen::add_clean_inode(CInode*)+0x121) [0x5640f2f49fc1]
6:
(Locker::adjust_cap_wanted(Capability*, int, int)+0x426) [0x5640f305e036]
7:
(Locker::process_request_cap_release(boost::intrusive_ptr<MDRequestImpl>&,
client_t, ceph_mds_request_release const&, std::basic_string_view<char,
std::char_traits<char> >)+0x599) [0x5640f307f7e9]
8:
(Server::handle_client_request(boost::intrusive_ptr<MClientRequest const>
const&)+0xc06) [0x5640f2f2a7c6]
9:
(Server::dispatch(boost::intrusive_ptr<Message const> const&)+0x13c)
[0x5640f2f2ef6c]
10:
(MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5db)
[0x5640f2e7727b]
11:
(MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const>
const&)+0x5c) [0x5640f2e778bc]
12:
(MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1bf)
[0x5640f2e60c2f]
13:
(Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478)
[0x7f3fe5c97ed8]
14: (DispatchQueue::entry()+0x50f)
[0x7f3fe5c9531f]
15:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe5d5f381]
16:
/lib64/libpthread.so.0(+0x81ca) [0x7f3fe4a0b1ca]
17: clone()
Deleting the file with cephfs-shell also does give Input/output error (5).
Does anyone have an idea on how to proceed here? I am perfectly fine
with loosing the affected files, they can all be easily restored from
backup. ceph config set mds debug_mds 20
ceph config set mds debug_ms 1
[1] https://docs.ceph.com/en/reef/man/8/ceph-post-file/
--
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
29 Nov
29 Nov
11:11 p.m.
Hello Patrick,
On 27.11.23 19:05, Patrick Donnelly wrote:
I would **really** love to see the debug logs from the MDS. Please
upload them using ceph-post-file [1]. If you can reliably reproduce,
turn on more debugging:
Uploaded debug log and core dump, see ceph-post-file:
02f78445-7136-44c9-a362-410de37a0b7d
Unfortunately, we cannot easily shut down normal access to the cluster
for these tests, therefore there is quite some clutter in the logs. The
logs show three crashes, the last one with enabled core dumping (ulimits
set to unlimited)
A note on reproducibility: To recreate the crash, reading the contents
of the file prior to removal seems necessary. Simply calling stat on the
file and then performing the removal also yields an Input/output error
but does not crash the MDS.
Interestingly, the MDS_DAMAGE flag is reset on restart of the MDS and
only comes back once the files in question are accessed (stat call is
sufficient).
For now, I'll hold off on running first-damage.py to try to remove the
affected files / inodes. Ultimately however, this seems to be the most
sensible solution to me, at least with regards to cluster downtime.
Cheers
Sebastian
ceph config set mds debug_mds 20
ceph config set mds debug_ms 1
[1] https://docs.ceph.com/en/reef/man/8/ceph-post-file/
30 Nov
30 Nov
5:58 a.m.
Hi Sebastian,
On Wed, Nov 29, 2023 at 3:11 PM Sebastian Knust
<sknust(a)physik.uni-bielefeld.de> wrote:
Hello Patrick,
On 27.11.23 19:05, Patrick Donnelly wrote:
I've not yet fully reviewed the logs but it seems there is a bug in
the detection logic which causes a spurious abort. This does not
appear to be actually new damage.
Are you using postgres? If you can share details about your snapshot
workflow and general workloads that would be helpful (privately if
desired).
I would **really** love to see the debug logs from the MDS. Please
upload them using ceph-post-file [1]. If you can reliably reproduce,
turn on more debugging:
Uploaded debug log and core dump, see ceph-post-file:
02f78445-7136-44c9-a362-410de37a0b7d
Unfortunately, we cannot easily shut down normal access to the cluster
for these tests, therefore there is quite some clutter in the logs. The
logs show three crashes, the last one with enabled core dumping (ulimits
set to unlimited)
A note on reproducibility: To recreate the crash, reading the contents
of the file prior to removal seems necessary. Simply calling stat on the
file and then performing the removal also yields an Input/output error
but does not crash the MDS.
Interestingly, the MDS_DAMAGE flag is reset on restart of the MDS and
only comes back once the files in question are accessed (stat call is
sufficient). ceph config set mds debug_mds 20
ceph config set mds debug_ms 1
[1] https://docs.ceph.com/en/reef/man/8/ceph-post-file/
For now, I'll hold off on running first-damage.py
to try to remove the
affected files / inodes. Ultimately however, this seems to be the most
sensible solution to me, at least with regards to cluster downtime.
Please give me another day to review then feel free to use
first-damage.py to cleanup. If you see new damage please upload the
logs.
--
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
5:38 p.m.
Hi Patrick,
On 30.11.23 03:58, Patrick Donnelly wrote:
We are in no hurry and will probably run first-damage.py sometime next
week. I will report new damage if it comes in.
Cheers
Sebastian
--
Dr. Sebastian Knust | Bielefeld University
IT Administrator | Faculty of Physics
Office: D2-110 | Universitätsstr. 25
Phone: +49 521 106 5234 | 33615 Bielefeld
I've not yet fully reviewed the logs but it seems there is a bug in
the detection logic which causes a spurious abort. This does not
appear to be actually new damage.
We are accessing the metadata (read-only) daily. The issue only popped
up after updating to 17.2.7. Of course, this does not mean that there
was no damage there before, only that it was not detected.
Are you using postgres?
Not on top of CephFS, no. We do use postgres on some RBD
volumes.
If you can share details about your snapshot
workflow and general workloads that would be helpful (privately if
desired).
Our CephFS root looks like this:
/archive
/homes
/no-snapshot
/other-snapshot
/scratch
We are running snapshots on /homes and /other-snapshot with the same
schedule. We mount the filesystem with a Kernel client on one of the
Ceph Hosts (not running the MDS) and mkdir / rmdir as needed.
- daily between 06:00 and 19:45 UTC (inclusive): Create a snapshot every
15 minutes, delete it unless it is hourly (xx:00) one hour later
- daily on the full hour: Create a snapshot, delete the 24 hours old
snapshot unless it is midnight
- daily at midnight delete the snapshot from 14 days ago unless it is Sunday
- every Sunday at midnight delete the snapshot from 8 weeks ago
Workload is two main Samba servers (one only sharing a subdirectory
which is generally not accessed on the other). Client access to those
servers is limited to 1GBit/s each. Until Tuesday, we also had a
mailserver with Dovecot running on top of CephFS. This was migrated on
Tuesday to an RBD volume as we had some issues with hanging access to
some files / directories (interestingly only in the main tree, in
snapshots access was without issue). Additionally, we have a Nextcloud
instance with ~200 active users storing data in CephFS as well as some
other Kernel clients with little / sporadic traffic, some running Samba,
some NFS, some interactive SSH / x2go servers with direct user access,
some specialised web applications (notably OMERO).
We run daily incremental backups of most of the CephFS content with
Bareos running on a dedicated server which has the whole CephFS tree
mounted read-only. For most data a full backup is performed every two
months, for some data only every six months. The affected area is
contained in this "every six months" full backup portion of the file
system tree.
Two weeks ago we deleted a folder structure with 6 TB, average file size
in the range of 1GB. The structure was unter /other-snapshot as well.
This led to severe load on the MDS, especially starting midnight. In
conjunction with Ubuntu kernel mount, we also had issues with
non-released capabilities preventing read-access to the /other-snapshot
part.
To combat these lingering problems, we deleted all snapshots in
/other-snapshot which led to a half a dozen PGs stuck in snaptrim state
(and a few hundred in snaptrim_wait). Updating from 17.2.6 to 17.2.7
solved that issue quickly, the affected PGs became unstuck and the whole
cluster was in active+clean a few hours later.
For now, I'll hold off on running
first-damage.py to try to remove the
affected files / inodes. Ultimately however, this seems to be the most
sensible solution to me, at least with regards to cluster downtime.
Please give me another day to review then feel free to use
first-damage.py to cleanup. If you see new damage please upload the
logs.
172
days inactive
178
days old
5 comments
3 participants
participants (3)
-
Dan van der Ster -
Patrick Donnelly -
Sebastian Knust