'MongoDB throwing "metadata corruption: files file:sizeStorer.wt and file:WiredTigerHS.wt have the same file ID 1"
My initial database was 3.2, and I spent the the time upgrading through every release to get to 4.4.0. Mongodb will no longer start when getting to 4.4.0, throwing a "metadata corruption" error. I'd hate to lose all of my test data. Is there any way to recover this?
{
"error":-31804,
"message":"[1598288597:93861][388261:0x7fe892bfec80], txn-recover: __recovery_setup_file, 432: metadata corruption: files file:sizeStorer.wt and file:WiredTigerHS.wt have the same file ID 1: WT_PANIC: WiredTiger library panic"
}
{"t":{"$date":"2020-08-24T13:03:16.588-04:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2020-08-24T13:03:16.591-04:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2020-08-24T13:03:16.591-04:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2020-08-24T13:03:16.591-04:00"},"s":"I", "c":"STORAGE", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":388261,"port":27017,"dbPath":"/home/myuser/Documents/development/data/mongo/data","architecture":"64-bit","host":"PCID075"}}
{"t":{"$date":"2020-08-24T13:03:16.591-04:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.0","gitVersion":"563487e100c4215e2dce98d0af2a6a5a2d67c5cf","openSSLVersion":"OpenSSL 1.1.1f 31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2020-08-24T13:03:16.591-04:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
{"t":{"$date":"2020-08-24T13:03:16.591-04:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"storage":{"dbPath":"/home/myUser/Documents/development/data/mongo/data","engine":"wiredTiger"}}}}
{"t":{"$date":"2020-08-24T13:03:16.592-04:00"},"s":"I", "c":"STORAGE", "id":22297, "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
{"t":{"$date":"2020-08-24T13:03:16.592-04:00"},"s":"I", "c":"STORAGE", "id":22315, "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=31581M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2020-08-24T13:03:16.982-04:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1598288596:982003][388261:0x7fe892bfec80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 162 through 164"}}
{"t":{"$date":"2020-08-24T13:03:17.011-04:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1598288597:11579][388261:0x7fe892bfec80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 163 through 164"}}
{"t":{"$date":"2020-08-24T13:03:17.043-04:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1598288597:43458][388261:0x7fe892bfec80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 164 through 164"}}
{"t":{"$date":"2020-08-24T13:03:17.093-04:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31804,"message":"[1598288597:93861][388261:0x7fe892bfec80], txn-recover: __recovery_setup_file, 432: metadata corruption: files file:sizeStorer.wt and file:WiredTigerHS.wt have the same file ID 1: WT_PANIC: WiredTiger library panic"}}
{"t":{"$date":"2020-08-24T13:03:17.093-04:00"},"s":"F", "c":"-", "id":23089, "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":50853,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp","line":446}}
{"t":{"$date":"2020-08-24T13:03:17.093-04:00"},"s":"F", "c":"-", "id":23090, "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
{"t":{"$date":"2020-08-24T13:03:17.093-04:00"},"s":"F", "c":"CONTROL", "id":4757800, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31431, "ctx":"initandlisten","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"55635C707811","b":"556359A4C000","o":"2CBB811","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"},{"a":"55635C708EB9","b":"556359A4C000","o":"2CBCEB9","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"55635C7066A6","b":"556359A4C000","o":"2CBA6A6","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},{"a":"7FE893ABF3C0","b":"7FE893AAA000","o":"153C0","s":"funlockfile","s+":"60"},{"a":"7FE8938FE18B","b":"7FE8938B8000","o":"4618B","s":"gsignal","s+":"CB"},{"a":"7FE8938DD859","b":"7FE8938B8000","o":"25859","s":"abort","s+":"12B"},{"a":"55635A9502F2","b":"556359A4C000","o":"F042F2","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"12B"},{"a":"55635A63FCC9","b":"556359A4C000","o":"BF3CC9","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1026","s+":"16"},{"a":"55635AB1C27B","b":"556359A4C000","o":"10D027B","s":"__eventv","s+":"3FB"},{"a":"55635A650B0E","b":"556359A4C000","o":"C04B0E","s":"__wt_panic_func","s+":"AA"},{"a":"55635A652565","b":"556359A4C000","o":"C06565","s":"__recovery_setup_file.cold.4","s+":"28"},{"a":"55635AB3A440","b":"556359A4C000","o":"10EE440","s":"__recovery_file_scan","s+":"B0"},{"a":"55635AB3AD41","b":"556359A4C000","o":"10EED41","s":"__wt_txn_recover","s+":"701"},{"a":"55635AAA2707","b":"556359A4C000","o":"1056707","s":"__wt_connection_workers","s+":"37"},{"a":"55635AA9B829","b":"556359A4C000","o":"104F829","s":"wiredtiger_open","s+":"24E9"},{"a":"55635AA4B3EA","b":"556359A4C000","o":"FFF3EA","s":"_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_","s+":"BA"},{"a":"55635AA52FBA","b":"556359A4C000","o":"1006FBA","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb","s+":"12DA"},{"a":"55635AA30F00","b":"556359A4C000","o":"FE4F00","s":"_ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createERKNS_19StorageGlobalParamsEPKNS_21StorageEngineLockFileE","s+":"120"},{"a":"55635B2F190E","b":"556359A4C000","o":"18A590E","s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE","s+":"4CE"},{"a":"55635A9ED507","b":"556359A4C000","o":"FA1507","s":"_ZN5mongo12_GLOBAL__N_114_initAndListenEPNS_14ServiceContextEi.isra.1409","s+":"3F7"},{"a":"55635A9F1590","b":"556359A4C000","o":"FA5590","s":"_ZN5mongo12_GLOBAL__N_111mongoDbMainEiPPcS2_","s+":"650"},{"a":"55635A9607C9","b":"556359A4C000","o":"F147C9","s":"main","s+":"9"},{"a":"7FE8938DF0B3","b":"7FE8938B8000","o":"270B3","s":"__libc_start_main","s+":"F3"},{"a":"55635A9EC9BE","b":"556359A4C000","o":"FA09BE","s":"_start","s+":"2E"}],"processInfo":{"mongodbVersion":"4.4.0","gitVersion":"563487e100c4215e2dce98d0af2a6a5a2d67c5cf","compiledModules":[],"uname":{"sysname":"Linux","release":"5.4.0-42-generic","version":"#46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020","machine":"x86_64"},"somap":[{"b":"556359A4C000","elfType":3,"buildId":"77B6A138746C90015067F12B963853BD51DAA5A6"},{"b":"7FE893AAA000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"775CBBFFF814456660786780B0B3B40096B4C05E"},{"b":"7FE8938B8000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"634252E0C5F8B03957A2E529719D4101699A894A"}]}}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635C707811","b":"556359A4C000","o":"2CBB811","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635C708EB9","b":"556359A4C000","o":"2CBCEB9","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635C7066A6","b":"556359A4C000","o":"2CBA6A6","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"7FE893ABF3C0","b":"7FE893AAA000","o":"153C0","s":"funlockfile","s+":"60"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"7FE8938FE18B","b":"7FE8938B8000","o":"4618B","s":"gsignal","s+":"CB"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"7FE8938DD859","b":"7FE8938B8000","o":"25859","s":"abort","s+":"12B"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635A9502F2","b":"556359A4C000","o":"F042F2","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"12B"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635A63FCC9","b":"556359A4C000","o":"BF3CC9","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1026","s+":"16"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635AB1C27B","b":"556359A4C000","o":"10D027B","s":"__eventv","s+":"3FB"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635A650B0E","b":"556359A4C000","o":"C04B0E","s":"__wt_panic_func","s+":"AA"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635A652565","b":"556359A4C000","o":"C06565","s":"__recovery_setup_file.cold.4","s+":"28"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635AB3A440","b":"556359A4C000","o":"10EE440","s":"__recovery_file_scan","s+":"B0"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635AB3AD41","b":"556359A4C000","o":"10EED41","s":"__wt_txn_recover","s+":"701"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635AAA2707","b":"556359A4C000","o":"1056707","s":"__wt_connection_workers","s+":"37"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635AA9B829","b":"556359A4C000","o":"104F829","s":"wiredtiger_open","s+":"24E9"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635AA4B3EA","b":"556359A4C000","o":"FFF3EA","s":"_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_","s+":"BA"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635AA52FBA","b":"556359A4C000","o":"1006FBA","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb","s+":"12DA"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635AA30F00","b":"556359A4C000","o":"FE4F00","s":"_ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createERKNS_19StorageGlobalParamsEPKNS_21StorageEngineLockFileE","s+":"120"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635B2F190E","b":"556359A4C000","o":"18A590E","s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE","s+":"4CE"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635A9ED507","b":"556359A4C000","o":"FA1507","s":"_ZN5mongo12_GLOBAL__N_114_initAndListenEPNS_14ServiceContextEi.isra.1409","s+":"3F7"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635A9F1590","b":"556359A4C000","o":"FA5590","s":"_ZN5mongo12_GLOBAL__N_111mongoDbMainEiPPcS2_","s+":"650"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635A9607C9","b":"556359A4C000","o":"F147C9","s":"main","s+":"9"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"7FE8938DF0B3","b":"7FE8938B8000","o":"270B3","s":"__libc_start_main","s+":"F3"}}}
{"t":{"$date":"2020-08-24T13:03:17.159-04:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"initandlisten","msg":" Frame: {frame}","attr":{"frame":{"a":"55635A9EC9BE","b":"556359A4C000","o":"FA09BE","s":"_start","s+":"2E"}}}
Attempting to repair fails:
mongod --storageEngine wiredTiger --dbpath ~/Documents/development/data/mongo/data --bind_ip_all --repair
Output:
{"t":{"$date":"2020-08-25T10:30:41.796-04:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2020-08-25T10:30:41.798-04:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2020-08-25T10:30:41.798-04:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2020-08-25T10:30:41.799-04:00"},"s":"I", "c":"STORAGE", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":428690,"port":27017,"dbPath":"/home/myUser/Documents/development/data/mongo/data","architecture":"64-bit","host":"PCID075"}}
{"t":{"$date":"2020-08-25T10:30:41.799-04:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.0","gitVersion":"563487e100c4215e2dce98d0af2a6a5a2d67c5cf","openSSLVersion":"OpenSSL 1.1.1f 31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2020-08-25T10:30:41.799-04:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
{"t":{"$date":"2020-08-25T10:30:41.799-04:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"repair":true,"storage":{"dbPath":"/home/myUser/Documents/development/data/mongo/data","engine":"wiredTiger"}}}}
{"t":{"$date":"2020-08-25T10:30:41.802-04:00"},"s":"I", "c":"STORAGE", "id":22297, "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
{"t":{"$date":"2020-08-25T10:30:41.802-04:00"},"s":"I", "c":"STORAGE", "id":22315, "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=31581M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2020-08-25T10:30:42.196-04:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1598365842:196259][428690:0x7f3ec7f81c80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 162 through 169"}}
{"t":{"$date":"2020-08-25T10:30:42.228-04:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1598365842:228141][428690:0x7f3ec7f81c80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 163 through 169"}}
{"t":{"$date":"2020-08-25T10:30:42.258-04:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1598365842:258421][428690:0x7f3ec7f81c80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 164 through 169"}}
{"t":{"$date":"2020-08-25T10:30:42.287-04:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1598365842:287000][428690:0x7f3ec7f81c80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 165 through 169"}}
{"t":{"$date":"2020-08-25T10:30:42.317-04:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1598365842:317838][428690:0x7f3ec7f81c80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 166 through 169"}}
{"t":{"$date":"2020-08-25T10:30:42.350-04:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1598365842:350301][428690:0x7f3ec7f81c80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 167 through 169"}}
{"t":{"$date":"2020-08-25T10:30:42.381-04:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1598365842:381651][428690:0x7f3ec7f81c80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 168 through 169"}}
{"t":{"$date":"2020-08-25T10:30:42.409-04:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1598365842:409231][428690:0x7f3ec7f81c80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 169 through 169"}}
{"t":{"$date":"2020-08-25T10:30:42.447-04:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31804,"message":"[1598365842:447560][428690:0x7f3ec7f81c80], txn-recover: __recovery_setup_file, 432: metadata corruption: files file:sizeStorer.wt and file:WiredTigerHS.wt have the same file ID 1: WT_PANIC: WiredTiger library panic"}}
{"t":{"$date":"2020-08-25T10:30:42.447-04:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31804,"message":"[1598365842:447592][428690:0x7f3ec7f81c80], txn-recover: __recovery_setup_file, 432: the process must exit and restart: WT_PANIC: WiredTiger library panic"}}
{"t":{"$date":"2020-08-25T10:30:42.447-04:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31804,"message":"[1598365842:447601][428690:0x7f3ec7f81c80], txn-recover: __wt_txn_recover, 852: Recovery failed: WT_PANIC: WiredTiger library panic"}}
{"t":{"$date":"2020-08-25T10:30:42.450-04:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":0,"message":"[1598365842:450015][428690:0x7f3ec7f81c80], connection: __wt_cache_destroy, 369: cache server: exiting with 4 pages in memory and 0 pages evicted"}}
{"t":{"$date":"2020-08-25T10:30:42.450-04:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":0,"message":"[1598365842:450036][428690:0x7f3ec7f81c80], connection: __wt_cache_destroy, 374: cache server: exiting with 33576 image bytes in memory"}}
{"t":{"$date":"2020-08-25T10:30:42.450-04:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":0,"message":"[1598365842:450043][428690:0x7f3ec7f81c80], connection: __wt_cache_destroy, 377: cache server: exiting with 34641 bytes in memory"}}
{"t":{"$date":"2020-08-25T10:30:43.117-04:00"},"s":"W", "c":"STORAGE", "id":22347, "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2020-08-25T10:30:43.117-04:00"},"s":"W", "c":"STORAGE", "id":22349, "ctx":"initandlisten","msg":"Attempting to salvage WiredTiger metadata"}
{"t":{"$date":"2020-08-25T10:30:43.464-04:00"},"s":"F", "c":"STORAGE", "id":50947, "ctx":"initandlisten","msg":"Failed to salvage WiredTiger metadata","attr":{"details":"-31804: WT_PANIC: WiredTiger library panic"}}
{"t":{"$date":"2020-08-25T10:30:43.464-04:00"},"s":"F", "c":"-", "id":23091, "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":50947,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":1115}}
{"t":{"$date":"2020-08-25T10:30:43.464-04:00"},"s":"F", "c":"-", "id":23092, "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
Solution 1:[1]
Turns out it's a bug in version 4.4:
https://jira.mongodb.org/browse/WT-6623
Thanks for reporting the issue. Having a look at the information provided it appears that a bug was introduced in WT-6374. On upgrade to 4.4, WiredTiger creates a file to track history, the file id of this file was incorrectly allocated and is overlapping with a preexisting file. Which results in the error seen.
We're working on a fix and will provide updates and respond with further steps.
Solution 2:[2]
Released version 4.4.1 and the problem is not and will not be solved.
https://jira.mongodb.org/browse/WT-6623
REMEDIATION AND WORKAROUNDS
The issue is fixed in MongoDB 4.4.1. To avoid this issue, avoid upgrading to MongoDB 4.4.0.
Replica set nodes that have already been impacted by this issue need to be fully re-synced. Standalone nodes must be recovered from backup.
Solution 3:[3]
Was able to repair the database with such issue by manually changing the version of WiredTigerHS.wt metadata using HEX-editor:
- Open WiredTiger.wt using any HEX-editor
- Find "id=1," which goes after "WiredTigerHS.wt" (I had 4 such occurrences, 3 were after sizeStorer.wt, and 1 after WiredTiger.wt)
- Change id to another value (9 worked for me).
- Try mongo repair
Restoring from a backup is more reliable, but if someone hadn't made a backup (like me, though it was a testing database anyway), this could be a workaround.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 | John Manko |
Solution 2 | |
Solution 3 | Monyak |