[ceph-users] How to recover from corrupted RocksDb

[Mario Giammarco]

Hello,
I have a ceph installation in a proxmox cluster.
Due to a temporary hardware glitch now I get this error on osd startup

-6> 2018-11-26 18:02:33.179327 7fa1d784be00 0 osd.0 1033 crush map has

features 1009089991638532096, adjusting msgr requires for osds
[/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1591]
Compaction error: Corruption: block checksum mismatch
-4> 2018-11-26 18:02:34.143123 7fa1c33f9700 4 rocksdb: (Original Log Time
2018/11/26-18:02:34.143021)
[/build/ceph-12.2.9/src/rocksdb/db/compaction_job.cc:621] [default]
compacted to: base level 1 max bytes base268435456 files[17$
-3> 2018-11-26 18:02:34.143126 7fa1c33f9700 4 rocksdb: (Original Log Time
2018/11/26-18:02:34.143068) EVENT_LOG_v1 {"time_micros": 1543251754143044,
1997048, "out$
[/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1275]
Waiting after background compaction error: Corruption: block checksum
mismatch, Accumulated background err$
submit_transaction error: Corruption: block checksum mismatch code = 2
Delete( Prefix = O key =
0x7f7ffffffffffffffb64000000217363'rub_3.26!='0xfffffffffffffffeffffffffffffffff'o')
Put( Prefix = S key = 'nid_max' Value size = 8)
Put( Prefix = S key = 'blobid_max' Value size = 8)
0> 2018-11-26 18:02:34.675641 7fa1c4bfc700 -1
/build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: In function 'void
BlueStore::_kv_sync_thread()' thread 7fa1c4bfc700 time 2018-11-26
18:02:34.674193
/build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: 8717: FAILED assert(r ==
0)
ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x55ec83876092]
2: (BlueStore::_kv_sync_thread()+0x24b5) [0x55ec836ffb55]
3: (BlueStore::KVSyncThread::entry()+0xd) [0x55ec8374040d]
4: (()+0x7494) [0x7fa1d5027494]
5: (clone()+0x3f) [0x7fa1d4098acf]

I have tried to recover it using ceph-bluestore-tool fsck and repair DEEP
but it says it is ALL ok.
I see that rocksd ldb tool needs .db files to recover and not a partition
so I cannot use it.
I do not understand why I cannot start osd if ceph-bluestore-tools says me
I have lost no data.
Can you help me?
Thanks,
Mario

[Wido den Hollander]

Post by Mario Giammarco
Hello,
I have a ceph installation in a proxmox cluster.
Due to a temporary hardware glitch now I get this error on osd startup
-6> 2018-11-26 18:02:33.179327 7fa1d784be00  0 osd.0 1033 crush map
has features 1009089991638532096, adjusting msgr requires for osds 
[/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1591]
Compaction error: Corruption: block checksum mismatch 
-4> 2018-11-26 18:02:34.143123 7fa1c33f9700 4 rocksdb: (Original Log
Time 2018/11/26-18:02:34.143021)
[/build/ceph-12.2.9/src/rocksdb/db/compaction_job.cc:621] [default]
compacted to: base level 1 max bytes base268435456 files[17$ 
                    
-3> 2018-11-26 18:02:34.143126 7fa1c33f9700 4 rocksdb: (Original Log
1543251754143044, "job": 3, "event": "compaction_finished",
"compaction_time_micros": 1997048, "out$ 
[/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1275]
Waiting after background compaction error: Corruption: block
checksum mismatch, Accumulated background err$ 
submit_transaction error: Corruption: block checksum mismatch code =
2 Rocksdb transaction: 
Delete( Prefix = O key =
0x7f7ffffffffffffffb64000000217363'rub_3.26!='0xfffffffffffffffeffffffffffffffff'o') 
Put( Prefix = S key = 'nid_max' Value size = 8) 
Put( Prefix = S key = 'blobid_max' Value size = 8) 
    0> 2018-11-26 18:02:34.675641 7fa1c4bfc700 -1
/build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: In function 'void
BlueStore::_kv_sync_thread()' thread 7fa1c4bfc700 time 2018-11-26
18:02:34.674193 
/build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: 8717: FAILED
assert(r == 0) 
                  
ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217)
luminous (stable) 
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x55ec83876092] 
2: (BlueStore::_kv_sync_thread()+0x24b5) [0x55ec836ffb55] 
3: (BlueStore::KVSyncThread::entry()+0xd) [0x55ec8374040d] 
4: (()+0x7494) [0x7fa1d5027494] 
5: (clone()+0x3f) [0x7fa1d4098acf]
I have tried to recover it using ceph-bluestore-tool fsck and repair
DEEP but it says it is ALL ok.
I see that rocksd ldb tool needs .db files to recover and not a
partition so I cannot use it.
I do not understand why I cannot start osd if ceph-bluestore-tools says
me I have lost no data.
Can you help me?

Why would you try to recover a individual OSD? If all your Placement
Groups are active(+clean) just wipe the OSD and re-deploy it.

What's the status of your PGs?

It says there is a checksum error (probably due to the hardware glitch)
so it refuses to start.

Don't try to outsmart Ceph, let backfill/recovery handle this. Trying to
manually fix this will only make things worse.

Wido

Post by Mario Giammarco
Thanks,
Mario
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Mario Giammarco]

I have only that copy, it is a showroom system but someone put a production
vm on it.

Post by Mario Giammarco
Hello,
I have a ceph installation in a proxmox cluster.
Due to a temporary hardware glitch now I get this error on osd startup
-6> 2018-11-26 18:02:33.179327 7fa1d784be00 0 osd.0 1033 crush map
has features 1009089991638532096, adjusting msgr requires for osds
[/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1591]
Compaction error: Corruption: block checksum mismatch
-4> 2018-11-26 18:02:34.143123 7fa1c33f9700 4 rocksdb: (Original Log
Time 2018/11/26-18:02:34.143021)
[/build/ceph-12.2.9/src/rocksdb/db/compaction_job.cc:621] [default]
compacted to: base level 1 max bytes base268435456 files[17$
-3> 2018-11-26 18:02:34.143126 7fa1c33f9700 4 rocksdb: (Original Log
1543251754143044, "job": 3, "event": "compaction_finished",
"compaction_time_micros": 1997048, "out$
[/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1275]
Waiting after background compaction error: Corruption: block
checksum mismatch, Accumulated background err$
submit_transaction error: Corruption: block checksum mismatch code =
Delete( Prefix = O key =

0x7f7ffffffffffffffb64000000217363'rub_3.26!='0xfffffffffffffffeffffffffffffffff'o')

Post by Mario Giammarco
Put( Prefix = S key = 'nid_max' Value size = 8)
Put( Prefix = S key = 'blobid_max' Value size = 8)
0> 2018-11-26 18:02:34.675641 7fa1c4bfc700 -1
/build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: In function 'void
BlueStore::_kv_sync_thread()' thread 7fa1c4bfc700 time 2018-11-26
18:02:34.674193
/build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: 8717: FAILED
assert(r == 0)
ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217)
luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x55ec83876092]
2: (BlueStore::_kv_sync_thread()+0x24b5) [0x55ec836ffb55]
3: (BlueStore::KVSyncThread::entry()+0xd) [0x55ec8374040d]
4: (()+0x7494) [0x7fa1d5027494]
5: (clone()+0x3f) [0x7fa1d4098acf]
I have tried to recover it using ceph-bluestore-tool fsck and repair
DEEP but it says it is ALL ok.
I see that rocksd ldb tool needs .db files to recover and not a
partition so I cannot use it.
I do not understand why I cannot start osd if ceph-bluestore-tools says
me I have lost no data.
Can you help me?

Why would you try to recover a individual OSD? If all your Placement
Groups are active(+clean) just wipe the OSD and re-deploy it.
What's the status of your PGs?
It says there is a checksum error (probably due to the hardware glitch)
so it refuses to start.
Don't try to outsmart Ceph, let backfill/recovery handle this. Trying to
manually fix this will only make things worse.
Wido

Post by Mario Giammarco
Thanks,
Mario
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Wido den Hollander]

Post by Mario Giammarco
I have only that copy, it is a showroom system but someone put a
production vm on it.

I have a feeling this won't be easy to fix or actually fixable:

- Compaction error: Corruption: block checksum mismatch
- submit_transaction error: Corruption: block checksum mismatch

RocksDB got corrupted on that OSD and won't be able to start now.

I wouldn't know where to start with this OSD.

Wido

Post by Mario Giammarco
Il giorno gio 29 nov 2018 alle ore 10:43 Wido den Hollander

Post by Mario Giammarco
Hello,
I have a ceph installation in a proxmox cluster.
Due to a temporary hardware glitch now I get this error on osd startup
     -6> 2018-11-26 18:02:33.179327 7fa1d784be00  0 osd.0 1033

crush map

Post by Mario Giammarco
     has features 1009089991638532096, adjusting msgr requires for

osds 

Post by Mario Giammarco
   

 [/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1591]

Post by Mario Giammarco
     Compaction error: Corruption: block checksum mismatch 

(Original Log

Post by Mario Giammarco
     Time 2018/11/26-18:02:34.143021)
     [/build/ceph-12.2.9/src/rocksdb/db/compaction_job.cc:621]

[default]

Post by Mario Giammarco
     compacted to: base level 1 max bytes base268435456 files[17$ 
                         

(Original Log

Post by Mario Giammarco
     1543251754143044, "job": 3, "event": "compaction_finished",
     "compaction_time_micros": 1997048, "out$ 
   

 [/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1275]

Post by Mario Giammarco
     Waiting after background compaction error: Corruption: block
     checksum mismatch, Accumulated background err$ 
     submit_transaction error: Corruption: block checksum mismatch

code =

Post by Mario Giammarco
     2 Rocksdb transaction: 
     Delete( Prefix = O key =
   

 0x7f7ffffffffffffffb64000000217363'rub_3.26!='0xfffffffffffffffeffffffffffffffff'o') 

Post by Mario Giammarco
     Put( Prefix = S key = 'nid_max' Value size = 8) 
     Put( Prefix = S key = 'blobid_max' Value size = 8) 
         0> 2018-11-26 18:02:34.675641 7fa1c4bfc700 -1
     /build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: In function

'void

Post by Mario Giammarco
     BlueStore::_kv_sync_thread()' thread 7fa1c4bfc700 time 2018-11-26
     18:02:34.674193 
     /build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: 8717: FAILED
     assert(r == 0) 
                       
     ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217)
     luminous (stable) 
     1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
     const*)+0x102) [0x55ec83876092] 
     2: (BlueStore::_kv_sync_thread()+0x24b5) [0x55ec836ffb55] 
     3: (BlueStore::KVSyncThread::entry()+0xd) [0x55ec8374040d] 
     4: (()+0x7494) [0x7fa1d5027494] 
     5: (clone()+0x3f) [0x7fa1d4098acf]
I have tried to recover it using ceph-bluestore-tool fsck and repair
DEEP but it says it is ALL ok.
I see that rocksd ldb tool needs .db files to recover and not a
partition so I cannot use it.
I do not understand why I cannot start osd if ceph-bluestore-tools

says

Post by Mario Giammarco
me I have lost no data.
Can you help me?

Why would you try to recover a individual OSD? If all your Placement
Groups are active(+clean) just wipe the OSD and re-deploy it.
What's the status of your PGs?
It says there is a checksum error (probably due to the hardware glitch)
so it refuses to start.
Don't try to outsmart Ceph, let backfill/recovery handle this. Trying to
manually fix this will only make things worse.
Wido

Post by Mario Giammarco
Thanks,
Mario
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Mario Giammarco]

The only strange thing is that ceph-bluestore-tool says that repair was
done, no errors are found and all is ok.
I ask myself what really does that tool.
Mario

Post by Wido den Hollander

Post by Mario Giammarco
I have only that copy, it is a showroom system but someone put a
production vm on it.

- Compaction error: Corruption: block checksum mismatch
- submit_transaction error: Corruption: block checksum mismatch
RocksDB got corrupted on that OSD and won't be able to start now.
I wouldn't know where to start with this OSD.
Wido

Post by Mario Giammarco
Il giorno gio 29 nov 2018 alle ore 10:43 Wido den Hollander

Post by Mario Giammarco
Hello,
I have a ceph installation in a proxmox cluster.
Due to a temporary hardware glitch now I get this error on osd

startup

Post by Mario Giammarco

Post by Mario Giammarco
-6> 2018-11-26 18:02:33.179327 7fa1d784be00 0 osd.0 1033

crush map

Post by Mario Giammarco
has features 1009089991638532096, adjusting msgr requires for

osds
[/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1591]

Post by Mario Giammarco
Compaction error: Corruption: block checksum mismatch

(Original Log

Post by Mario Giammarco
Time 2018/11/26-18:02:34.143021)
[/build/ceph-12.2.9/src/rocksdb/db/compaction_job.cc:621]

[default]

Post by Mario Giammarco
compacted to: base level 1 max bytes base268435456 files[17$

(Original Log

Post by Mario Giammarco
1543251754143044, "job": 3, "event": "compaction_finished",
"compaction_time_micros": 1997048, "out$

[/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1275]

Post by Mario Giammarco
Waiting after background compaction error: Corruption: block
checksum mismatch, Accumulated background err$
submit_transaction error: Corruption: block checksum mismatch

code =

Post by Mario Giammarco
Delete( Prefix = O key =

0x7f7ffffffffffffffb64000000217363'rub_3.26!='0xfffffffffffffffeffffffffffffffff'o')

Post by Mario Giammarco

Post by Mario Giammarco
Put( Prefix = S key = 'nid_max' Value size = 8)
Put( Prefix = S key = 'blobid_max' Value size = 8)
0> 2018-11-26 18:02:34.675641 7fa1c4bfc700 -1
/build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: In function

'void

Post by Mario Giammarco
BlueStore::_kv_sync_thread()' thread 7fa1c4bfc700 time

2018-11-26

Post by Mario Giammarco

Post by Mario Giammarco
18:02:34.674193
/build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: 8717: FAILED
assert(r == 0)
ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217)
luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int,

char

Post by Mario Giammarco

Post by Mario Giammarco
const*)+0x102) [0x55ec83876092]
2: (BlueStore::_kv_sync_thread()+0x24b5) [0x55ec836ffb55]
3: (BlueStore::KVSyncThread::entry()+0xd) [0x55ec8374040d]
4: (()+0x7494) [0x7fa1d5027494]
5: (clone()+0x3f) [0x7fa1d4098acf]
I have tried to recover it using ceph-bluestore-tool fsck and

repair

Post by Mario Giammarco

Post by Mario Giammarco
DEEP but it says it is ALL ok.
I see that rocksd ldb tool needs .db files to recover and not a
partition so I cannot use it.
I do not understand why I cannot start osd if ceph-bluestore-tools

says

Post by Mario Giammarco
me I have lost no data.
Can you help me?

Why would you try to recover a individual OSD? If all your Placement
Groups are active(+clean) just wipe the OSD and re-deploy it.
What's the status of your PGs?
It says there is a checksum error (probably due to the hardware

glitch)

Post by Mario Giammarco
so it refuses to start.
Don't try to outsmart Ceph, let backfill/recovery handle this.

Trying to

Post by Mario Giammarco
manually fix this will only make things worse.
Wido

Post by Mario Giammarco
Thanks,
Mario
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Igor Fedotov]

'ceph-bluestore-tool repair' checks and repairs BlueStore metadata
consistency not RocksDB one.

It looks like you're observing CRC mismatch during DB compaction which
is probably not triggered during the repair.

Good point is that it looks like Bluestore's metadata are consistent and
hence data recovery is still possible  - potentially, can't build up a
working procedure using existing tools though..

Let me check if one can disable DB compaction using rocksdb settings.

Post by Mario Giammarco
The only strange thing is that ceph-bluestore-tool says that repair
was done, no errors are found and all is ok.
I ask myself what really does that tool.
Mario
Il giorno gio 29 nov 2018 alle ore 11:03 Wido den Hollander

Post by Mario Giammarco
I have only that copy, it is a showroom system but someone put a
production vm on it.

- Compaction error: Corruption: block checksum mismatch
- submit_transaction error: Corruption: block checksum mismatch
RocksDB got corrupted on that OSD and won't be able to start now.
I wouldn't know where to start with this OSD.
Wido

Post by Mario Giammarco
Il giorno gio 29 nov 2018 alle ore 10:43 Wido den Hollander
     > Hello,
     > I have a ceph installation in a proxmox cluster.
     > Due to a temporary hardware glitch now I get this error on

osd startup

Post by Mario Giammarco
     >
     >     -6> 2018-11-26 18:02:33.179327 7fa1d784be00  0 osd.0 1033
     crush map
     >     has features 1009089991638532096, adjusting msgr

requires for

Post by Mario Giammarco
     osds
     >

  [/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1591]

Post by Mario Giammarco
     >     Compaction error: Corruption: block checksum mismatch
     (Original Log
     >     Time 2018/11/26-18:02:34.143021)
     >  [/build/ceph-12.2.9/src/rocksdb/db/compaction_job.cc:621]
     [default]
     >     compacted to: base level 1 max bytes

base268435456 files[17$

Post by Mario Giammarco
     >
     (Original Log
     >     Time 2018/11/26-18:02:34.143068) EVENT_LOG_v1

"compaction_finished",

Post by Mario Giammarco
     >     "compaction_time_micros": 1997048, "out$
     >

  [/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1275]
block

Post by Mario Giammarco
     >     checksum mismatch, Accumulated background err$
     >     submit_transaction error: Corruption: block checksum

mismatch

Post by Mario Giammarco
     code =
     >     Delete( Prefix = O key =
     >

  0x7f7ffffffffffffffb64000000217363'rub_3.26!='0xfffffffffffffffeffffffffffffffff'o')

Post by Mario Giammarco
     >     Put( Prefix = S key = 'nid_max' Value size = 8)
     >     Put( Prefix = S key = 'blobid_max' Value size = 8)
     >         0> 2018-11-26 18:02:34.675641 7fa1c4bfc700 -1
     >  /build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: In function
     'void
     >     BlueStore::_kv_sync_thread()' thread 7fa1c4bfc700 time

2018-11-26

Post by Mario Giammarco
     >     18:02:34.674193

FAILED

Post by Mario Giammarco
     >     assert(r == 0)
     >
     >     ceph version 12.2.9

(9e300932ef8a8916fb3fda78c58691a6ab0f4217)

Post by Mario Giammarco
     >     luminous (stable)
     >     1: (ceph::__ceph_assert_fail(char const*, char const*,

int, char

Post by Mario Giammarco
     >     const*)+0x102) [0x55ec83876092]
     >     2: (BlueStore::_kv_sync_thread()+0x24b5) [0x55ec836ffb55]
     >     3: (BlueStore::KVSyncThread::entry()+0xd)

[0x55ec8374040d]

Post by Mario Giammarco
     >     4: (()+0x7494) [0x7fa1d5027494]
     >     5: (clone()+0x3f) [0x7fa1d4098acf]
     >
     >
     > I have tried to recover it using ceph-bluestore-tool fsck

and repair

Post by Mario Giammarco
     > DEEP but it says it is ALL ok.
     > I see that rocksd ldb tool needs .db files to recover and

not a

Post by Mario Giammarco
     > partition so I cannot use it.
     > I do not understand why I cannot start osd if

ceph-bluestore-tools

Post by Mario Giammarco
     says
     > me I have lost no data.
     > Can you help me?
     Why would you try to recover a individual OSD? If all your

Placement

Post by Mario Giammarco
     Groups are active(+clean) just wipe the OSD and re-deploy it.
     What's the status of your PGs?
     It says there is a checksum error (probably due to the

hardware glitch)

Post by Mario Giammarco
     so it refuses to start.
     Don't try to outsmart Ceph, let backfill/recovery handle

this. Trying to

Post by Mario Giammarco
     manually fix this will only make things worse.
     Wido
     > Thanks,
     > Mario
     >
     > _______________________________________________
     > ceph-users mailing list
     > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
     >

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Paul Emmerich]

does objectstore-tool still work? If yes:

export all the PGs on the OSD with objectstore-tool and important them
into a new OSD.

Paul
--
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: [phone]

'ceph-bluestore-tool repair' checks and repairs BlueStore metadata consistency not RocksDB one.
It looks like you're observing CRC mismatch during DB compaction which is probably not triggered during the repair.
Good point is that it looks like Bluestore's metadata are consistent and hence data recovery is still possible - potentially, can't build up a working procedure using existing tools though..
Let me check if one can disable DB compaction using rocksdb settings.
The only strange thing is that ceph-bluestore-tool says that repair was done, no errors are found and all is ok.
I ask myself what really does that tool.
Mario

Post by Wido den Hollander

Post by Mario Giammarco
I have only that copy, it is a showroom system but someone put a
production vm on it.

- Compaction error: Corruption: block checksum mismatch
- submit_transaction error: Corruption: block checksum mismatch
RocksDB got corrupted on that OSD and won't be able to start now.
I wouldn't know where to start with this OSD.
Wido

Post by Mario Giammarco
Il giorno gio 29 nov 2018 alle ore 10:43 Wido den Hollander

Post by Mario Giammarco
Hello,
I have a ceph installation in a proxmox cluster.
Due to a temporary hardware glitch now I get this error on osd startup
-6> 2018-11-26 18:02:33.179327 7fa1d784be00 0 osd.0 1033

crush map

Post by Mario Giammarco
has features 1009089991638532096, adjusting msgr requires for

osds
[/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1591]

Post by Mario Giammarco
Compaction error: Corruption: block checksum mismatch

(Original Log

Post by Mario Giammarco
Time 2018/11/26-18:02:34.143021)
[/build/ceph-12.2.9/src/rocksdb/db/compaction_job.cc:621]

[default]

Post by Mario Giammarco
compacted to: base level 1 max bytes base268435456 files[17$

(Original Log

Post by Mario Giammarco
1543251754143044, "job": 3, "event": "compaction_finished",
"compaction_time_micros": 1997048, "out$

[/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1275]

Post by Mario Giammarco
Waiting after background compaction error: Corruption: block
checksum mismatch, Accumulated background err$
submit_transaction error: Corruption: block checksum mismatch

code =

Post by Mario Giammarco
Delete( Prefix = O key =

0x7f7ffffffffffffffb64000000217363'rub_3.26!='0xfffffffffffffffeffffffffffffffff'o')

Post by Mario Giammarco
Put( Prefix = S key = 'nid_max' Value size = 8)
Put( Prefix = S key = 'blobid_max' Value size = 8)
0> 2018-11-26 18:02:34.675641 7fa1c4bfc700 -1
/build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: In function

'void

Post by Mario Giammarco
BlueStore::_kv_sync_thread()' thread 7fa1c4bfc700 time 2018-11-26
18:02:34.674193
/build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: 8717: FAILED
assert(r == 0)
ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217)
luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x55ec83876092]
2: (BlueStore::_kv_sync_thread()+0x24b5) [0x55ec836ffb55]
3: (BlueStore::KVSyncThread::entry()+0xd) [0x55ec8374040d]
4: (()+0x7494) [0x7fa1d5027494]
5: (clone()+0x3f) [0x7fa1d4098acf]
I have tried to recover it using ceph-bluestore-tool fsck and repair
DEEP but it says it is ALL ok.
I see that rocksd ldb tool needs .db files to recover and not a
partition so I cannot use it.
I do not understand why I cannot start osd if ceph-bluestore-tools

says

Post by Mario Giammarco
me I have lost no data.
Can you help me?

Why would you try to recover a individual OSD? If all your Placement
Groups are active(+clean) just wipe the OSD and re-deploy it.
What's the status of your PGs?
It says there is a checksum error (probably due to the hardware glitch)
so it refuses to start.
Don't try to outsmart Ceph, let backfill/recovery handle this. Trying to
manually fix this will only make things worse.
Wido

Post by Mario Giammarco
Thanks,
Mario
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Igor Fedotov]

Yeah, that may be the way.

Preferably to disable compaction during this procedure though.

To do that please set

bluestore rocksdb options = "disable_auto_compactions=true"

in [osd] section in ceph.conf


Thanks,

Igor

Post by Paul Emmerich
export all the PGs on the OSD with objectstore-tool and important them
into a new OSD.
Paul

[Paul Emmerich]

If this is really the last copy of important data: consider making a
full raw clone of the disk before running any ceph-objectstore-tool
commands on it and consider getting some professional help if you are
not too familiar with the inner workings of Ceph.

That being said, it's basically just:

ceph-objectstore-tool --bluestore --op export --pg <pgid> --data-path
/var/lib/ceph/osd/ceph-XX --file <pgid>

and --op import into the new OSD. I don't think there's a command for
all PGs at once, so you'll have to iterate over the broken PGs (or
--op list-pgs)

good luck.


Paul
--
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: [phone]

Am Do., 29. Nov. 2018 um 15:36 Uhr schrieb Davide Accetturi

Hi guys
Thanks for your replies,
What’s the right syntax of Ceph-objectstore-tool in order to export all of the PGs??
Thanks you so much
Davide
Sent from my iPhone

Post by Igor Fedotov
Yeah, that may be the way.
Preferably to disable compaction during this procedure though.
To do that please set
bluestore rocksdb options = "disable_auto_compactions=true"
in [osd] section in ceph.conf
Thanks,
Igor

Post by Paul Emmerich
export all the PGs on the OSD with objectstore-tool and important them
into a new OSD.
Paul