'How to display the memory maximum limit allowed for clickhouse server
I run a clickhouse server on a container having 30Gb Memory:
top
KiB Mem : 31746192 total, 958808 free, 11474896 used, 19312488 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 19581348 avail Mem
PID | USER | PR | NI | VIRT | RES | SHR | S | %CPU | %MEM | TIME+ | COMMAND |
---|---|---|---|---|---|---|---|---|---|---|---|
1 | clickho+ | 20 | 0 | 12.8g | 837508 | 357148 | S | 6.7 | 2.6 | 20:35.39 | clickhouse-serv |
I had some errors with background merges because it was reaching the 853Mb limit. Therefore, I changed the configuration like adding thinking that the default conf does not automatically know the available memory amount.
<clickhouse>
<max_server_memory_usage_to_ram_ratio>30</max_server_memory_usage_to_ram_ratio>
</clickhouse>
I would like to know how to display the current memory limit for background merges
and if there is a way to set this limit.
SELECT
metric,
':',
formatReadableSize(value)
FROM system.metrics
WHERE metric LIKE 'MemoryTracking'
┌─metric─────────┬─':'─┬─formatReadableSize(value)─┐
│ MemoryTracking │ : │ 815.71 MiB │
└────────────────┴─────┴───────────────────────────┘
The Memory error I originaly had was:
2022.05.06 07:29:53.203566 [ 323 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 860.17 MiB (attempt to allocate chunk of 4219892 bytes), maximum: 858.30 MiB. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xaebee1a in /usr/bin/clickhouse
1. DB::Exception::Exception<char const*, char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, long&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*&&, char const*&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, long&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) @ 0xaed6e0c in /usr/bin/clickhouse
2. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xaed6a04 in /usr/bin/clickhouse
3. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xaed6429 in /usr/bin/clickhouse
4. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xaed6429 in /usr/bin/clickhouse
5. DB::Memory<Allocator<false, false> >::alloc() @ 0xaf099ad in /usr/bin/clickhouse
6. DB::MergeTreeDataPartWriterOnDisk::Stream::Stream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::IDisk>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::ICompressionCodec> const&, unsigned long) @ 0x154face5 in /usr/bin/clickhouse
7. ? @ 0x15504ded in /usr/bin/clickhouse
8. DB::ISerialization::enumerateStreams(DB::ISerialization::SubstreamPath&, std::__1::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x14424ef1 in /usr/bin/clickhouse
9. DB::SerializationNullable::enumerateStreams(DB::ISerialization::SubstreamPath&, std::__1::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x144500a7 in /usr/bin/clickhouse
10. DB::ISerialization::enumerateStreams(DB::ISerialization::SubstreamPath&, std::__1::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::__1::shared_ptr<DB::IDataType const> const&) const @ 0x14425455 in /usr/bin/clickhouse
11. DB::MergeTreeDataPartWriterWide::addStreams(DB::NameAndTypePair const&, std::__1::shared_ptr<DB::IAST> const&) @ 0x154fe5a1 in /usr/bin/clickhouse
12. DB::MergeTreeDataPartWriterWide::MergeTreeDataPartWriterWide(std::__1::shared_ptr<DB::IMergeTreeDataPart const> const&, DB::NamesAndTypesList const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeIndex const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeIndex const> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::ICompressionCodec> const&, DB::MergeTreeWriterSettings const&, DB::MergeTreeIndexGranularity const&) @ 0x154fe42a in /usr/bin/clickhouse
13. DB::MergeTreeDataPartWide::getWriter(DB::NamesAndTypesList const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeIndex const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeIndex const> > > const&, std::__1::shared_ptr<DB::ICompressionCodec> const&, DB::MergeTreeWriterSettings const&, DB::MergeTreeIndexGranularity const&) const @ 0x154ef5e2 in /usr/bin/clickhouse
14. DB::MergedBlockOutputStream::MergedBlockOutputStream(std::__1::shared_ptr<DB::IMergeTreeDataPart const> const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::NamesAndTypesList const&, std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeIndex const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeIndex const> > > const&, std::__1::shared_ptr<DB::ICompressionCodec>, bool, bool) @ 0x155d074c in /usr/bin/clickhouse
15. void std::__1::allocator_traits<std::__1::allocator<DB::MergedBlockOutputStream> >::__construct<DB::MergedBlockOutputStream, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const>&, DB::NamesAndTypesList&, std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeIndex const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeIndex const> > >, std::__1::shared_ptr<DB::ICompressionCodec>&, bool, bool&>(std::__1::integral_constant<bool, true>, std::__1::allocator<DB::MergedBlockOutputStream>&, DB::MergedBlockOutputStream*, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const>&, DB::NamesAndTypesList&, std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeIndex const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeIndex const> > >&&, std::__1::shared_ptr<DB::ICompressionCodec>&, bool&&, bool&) @ 0x1544089e in /usr/bin/clickhouse
16. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() @ 0x15435bb8 in /usr/bin/clickhouse
17. bool std::__1::__function::__policy_invoker<bool ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::MergeTask::ExecuteAndFinalizeHorizontalPart::subtasks::'lambda'(), bool ()> >(std::__1::__function::__policy_storage const*) @ 0x15443d29 in /usr/bin/clickhouse
18. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::execute() @ 0x1543a56b in /usr/bin/clickhouse
19. DB::MergeTask::execute() @ 0x1543f51a in /usr/bin/clickhouse
20. DB::MergePlainMergeTreeTask::executeStep() @ 0x1543110c in /usr/bin/clickhouse
21. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>) @ 0xae960eb in /usr/bin/clickhouse
22. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::threadFunction() @ 0xae95d39 in /usr/bin/clickhouse
23. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xaf6556a in /usr/bin/clickhouse
24. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xaf675a4 in /usr/bin/clickhouse
25. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xaf62937 in /usr/bin/clickhouse
26. ? @ 0xaf663fd in /usr/bin/clickhouse
27. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
28. __clone @ 0xfeb0d in /usr/lib64/libc-2.17.so
but also connecting to clickhouse-client
I get
Cannot load data for command line suggestions: Code: 241. DB::Exception: Received from localhost:9001. DB::Exception: Memory limit (total) exceeded: would use 935.91 MiB (attempt to allocate chunk of 4440942 bytes), maximum: 858.30 MiB. (MEMORY_LIMIT_EXCEEDED) (version 22.2.3.1)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|