What's the right cmake command to cross-compile for Raspberry Pi from M1 Mac?

Question

My title seems general but I need to compile a project for my Raspberry Pi 3b. I want to use Telegram's Bot API server on it, but it takes a lot of time to compile, so I'm looking for faster ways to compile it. I heard about cross compiling, and I tried to compile the project from a Debian VM using UTM on my M1 Mac, but obviously arm64 isn't the same as armhf.

The main commands I need to perform are (from the project's page, selecting Debian 10+):

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=.. ..
cmake --build . --target install

from build folder.

I'm not an expert with compilation commands, and I've looked over a bunch of potential answer over the internet, but it's or too complicated, or not for cmake, or not for M1 Mac...

Anyway, I found what seems to be a pretty decent solution, however once again it's for clang++ and not cmake.

So my question is straightforward, what cmake arguments/commands can I use to easily build the binary for my (slow) Raspberry from my (fast) M1 Mac?

---

EDIT 1:
According to the answer of @sweenish, it's possible to pass directly arguments to the cmake command, which I did, wrapping them all into -DCMAKE_CXX_FLAGS.
My current command is:

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=.. -DCMAKE_CXX_FLAGS="--target=arm-linux-gnueabihf --sysroot $HOME/sysroot -isysroot $HOME/sysroot -isystem $HOME/sysroot/usr/include/c++/10 -isystem $HOME/sysroot/usr/include/arm-linux-gnueabihf/c++/10 -L$HOME/sysroot/usr/lib/gcc/arm-linux-gnueabihf/10 -B$HOME/sysroot/usr/lib/gcc/arm-linux-gnueabihf/10 --gcc-toolchain=$(brew --prefix arm-linux-gnueabihf-binutils)" ..

However, as the output shows, it seems like macOS/Xcode is natively appending 'default' arguments at the end of my command:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ --target=arm-linux-gnueabihf --sysroot /Users/antoine/sysroot -isysroot /Users/antoine/sysroot -isystem /Users/antoine/sysroot/usr/include/c++/10 -isystem /Users/antoine/sysroot/usr/include/arm-linux-gnueabihf/c++/10 -L/Users/antoine/sysroot/usr/lib/gcc/arm-linux-gnueabihf/10 -B/Users/antoine/sysroot/usr/lib/gcc/arm-linux-gnueabihf/10 --gcc-toolchain=/opt/homebrew/opt/arm-linux-gnueabihf-binutils  -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/cmTC_42ace.dir/testCXXCompiler.cxx.o -o cmTC_42ace

(notice the additional space after the end of my args and the beginning of uncontrolled args)

This leads to 3 main errors:

• -L$HOME... is not used
• -arch arm64, which I didn't choose to append, is unused too
• Error: unable to disambiguate: -search_paths_first (did you mean --search_paths_first ?), again in the appended args

So to sum up, 'disabling' system args should solve 2/3 of my problems, and removing the -L one should be enough to be finally able to compile. How do I get rid of these final appended arguments?

---