DllNotFoundException libgurobi95 when invoking lambda function on AWS

Question

I have a lambda function that uses the OPTANO.Modeling.Gurobi NuGet package (and OPTANO.Modeling).

The code works perfectly when I run it locally using the Mock Lambda Test Tool, but when I publish it to AWS and invoke it, it returns an error depending on how I do it.

If I publish via 'Publish to AWS Lambda...' in Visual Studio I get the following:

Unhandled exception. System.DllNotFoundException: Unable to load shared library 'gurobi95' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libgurobi95: cannot open shared object file: No such file or directory
at Gurobi.GurobiDLL.GRBloadenvadv(IntPtr& env, String logfilename, Int32 apitype, Int32 major, Int32 minor, Int32 tech, String server, String router, String password, String group, Int32 priority, Int32 idletimeout, String cloudid, String cloudkey, cbfunc callback, IntPtr usrdata, logcbfunc logcallback, IntPtr logdata)
at Gurobi.GRBEnv..ctor(String logfilename)
at Gurobi.GRBEnv..ctor()
at OPTANO.Modeling.Optimization.Solver.Gurobi950.GurobiSolver.InitializeEnvironment(GurobiSolverConfiguration configuration)
at OPTANO.Modeling.Optimization.Solver.Gurobi950.GurobiSolver..ctor(GurobiSolverConfiguration configuration)
at Testbed_Primary.Function.FunctionHandler(Boolean writeReportToConsole, Boolean writeWeightsToConsole) in xxxx\Function.cs:line 482
at Testbed_Primary.Function.CalculateStuff() in xxxx\Function.cs:line 118
at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__139_1(Object state)
at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
05 May 2022 21:31:49,146 [WARN] ([email protected]:331 errno: None) run_dotnet(dotnet_path, &args) failed

The problem seems to be related to gurobi95.dll. Sometimes it appears in the zip file, sometimes it doesn't. If I make sure it is in the zip file and instead use 'Upload from...' on AWS Console I get the following instead of the stack trace above. There's no more detail in the Cloudwatch log group.

Lambda encountered an UnhandledException via 'AppDomain.CurrentDomain.UnhandledException' event:

It appears to happen at the same point in the code. There is nothing after the colon. I would suggest that there's something wrong with the DLL and that AWS dumps it if uploading using the first method but not with the second, were it not for the fact that it works fine when run locally.

I've spent ages fiddling around with DLLs and settings, and searching for help on DLL-not-found errors, but I'm flummoxed.

Edit: Setting the LD_DEBUG environment variable to 'all cat' outputs the following to the Cloudwatch log. I don't know this helps me diagnose the problem though as it doesn't appear to indicate any errors. I imagine a lot of it is irrelevant.

10: symbol=_ZSt8__copy_nIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEmPcET1_T_T0_SC_St26random_access_iterator_tag;  lookup in file=dotnet [0]
10: symbol=_ZSt8__copy_nIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEmPcET1_T_T0_SC_St26random_access_iterator_tag;  lookup in file=/lib64/libpthread.so.0 [0]
10: symbol=_ZSt8__copy_nIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEmPcET1_T_T0_SC_St26random_access_iterator_tag;  lookup in file=/lib64/libdl.so.2 [0]
10: symbol=_ZSt8__copy_nIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEmPcET1_T_T0_SC_St26random_access_iterator_tag;  lookup in file=/lib64/libstdc++.so.6 [0]
10: symbol=_ZSt8__copy_nIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEmPcET1_T_T0_SC_St26random_access_iterator_tag;  lookup in file=/lib64/libm.so.6 [0]
10: symbol=_ZSt8__copy_nIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEmPcET1_T_T0_SC_St26random_access_iterator_tag;  lookup in file=/lib64/libgcc_s.so.1 [0]
10: symbol=_ZSt8__copy_nIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEmPcET1_T_T0_SC_St26random_access_iterator_tag;  lookup in file=/lib64/libc.so.6 [0]
10: symbol=_ZSt8__copy_nIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEmPcET1_T_T0_SC_St26random_access_iterator_tag;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
10: symbol=_ZSt8__copy_nIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEmPcET1_T_T0_SC_St26random_access_iterator_tag;  lookup in file=/var/runtime/liblambdaruntime.so [0]
10: binding file /var/runtime/liblambdaruntime.so [0] to /var/runtime/liblambdaruntime.so [0]: normal symbol `_ZSt8__copy_nIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEmPcET1_T_T0_SC_St26random_access_iterator_tag'
10: symbol=_ZSt4copyIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET0_T_SD_SC_;  lookup in file=dotnet [0]
10: symbol=_ZSt4copyIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET0_T_SD_SC_;  lookup in file=/lib64/libpthread.so.0 [0]
10: symbol=_ZSt4copyIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET0_T_SD_SC_;  lookup in file=/lib64/libdl.so.2 [0]
10: symbol=_ZSt4copyIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET0_T_SD_SC_;  lookup in file=/lib64/libstdc++.so.6 [0]
10: symbol=_ZSt4copyIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET0_T_SD_SC_;  lookup in file=/lib64/libm.so.6 [0]
10: symbol=_ZSt4copyIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET0_T_SD_SC_;  lookup in file=/lib64/libgcc_s.so.1 [0]
10: symbol=_ZSt4copyIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET0_T_SD_SC_;  lookup in file=/lib64/libc.so.6 [0]
10: symbol=_ZSt4copyIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET0_T_SD_SC_;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
10: symbol=_ZSt4copyIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET0_T_SD_SC_;  lookup in file=/var/runtime/liblambdaruntime.so [0]
10: binding file /var/runtime/liblambdaruntime.so [0] to /var/runtime/liblambdaruntime.so [0]: normal symbol `_ZSt4copyIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET0_T_SD_SC_'
10: symbol=_ZSt12__miter_baseIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEET_SB_;  lookup in file=dotnet [0]
10: symbol=_ZSt12__miter_baseIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEET_SB_;  lookup in file=/lib64/libpthread.so.0 [0]
10: symbol=_ZSt12__miter_baseIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEET_SB_;  lookup in file=/lib64/libdl.so.2 [0]
10: symbol=_ZSt12__miter_baseIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEET_SB_;  lookup in file=/lib64/libstdc++.so.6 [0]
10: symbol=_ZSt12__miter_baseIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEET_SB_;  lookup in file=/lib64/libm.so.6 [0]
10: symbol=_ZSt12__miter_baseIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEET_SB_;  lookup in file=/lib64/libgcc_s.so.1 [0]
10: symbol=_ZSt12__miter_baseIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEET_SB_;  lookup in file=/lib64/libc.so.6 [0]
10: symbol=_ZSt12__miter_baseIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEET_SB_;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
10: symbol=_ZSt12__miter_baseIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEET_SB_;  lookup in file=/var/runtime/liblambdaruntime.so [0]
10: binding file /var/runtime/liblambdaruntime.so [0] to /var/runtime/liblambdaruntime.so [0]: normal symbol `_ZSt12__miter_baseIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEET_SB_'
10: symbol=_ZSt14__copy_move_a2ILb0EN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET1_T0_SD_SC_;  lookup in file=dotnet [0]
10: symbol=_ZSt14__copy_move_a2ILb0EN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET1_T0_SD_SC_;  lookup in file=/lib64/libpthread.so.0 [0]
10: symbol=_ZSt14__copy_move_a2ILb0EN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET1_T0_SD_SC_;  lookup in file=/lib64/libdl.so.2 [0]
10: symbol=_ZSt14__copy_move_a2ILb0EN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET1_T0_SD_SC_;  lookup in file=/lib64/libstdc++.so.6 [0]
10: symbol=_ZSt14__copy_move_a2ILb0EN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET1_T0_SD_SC_;  lookup in file=/lib64/libm.so.6 [0]
10: symbol=_ZSt14__copy_move_a2ILb0EN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET1_T0_SD_SC_;  lookup in file=/lib64/libgcc_s.so.1 [0]
10: symbol=_ZSt14__copy_move_a2ILb0EN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET1_T0_SD_SC_;  lookup in file=/lib64/libc.so.6 [0]
10: symbol=_ZSt14__copy_move_a2ILb0EN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET1_T0_SD_SC_;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
10: symbol=_ZSt14__copy_move_a2ILb0EN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET1_T0_SD_SC_;  lookup in file=/var/runtime/liblambdaruntime.so [0]
10: binding file /var/runtime/liblambdaruntime.so [0] to /var/runtime/liblambdaruntime.so [0]: normal symbol `_ZSt14__copy_move_a2ILb0EN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPcET1_T0_SD_SC_'
10: symbol=_ZSt12__niter_baseIPcET_S1_;  lookup in file=dotnet [0]
10: symbol=_ZSt12__niter_baseIPcET_S1_;  lookup in file=/lib64/libpthread.so.0 [0]
10: symbol=_ZSt12__niter_baseIPcET_S1_;  lookup in file=/lib64/libdl.so.2 [0]
10: symbol=_ZSt12__niter_baseIPcET_S1_;  lookup in file=/lib64/libstdc++.so.6 [0]
10: symbol=_ZSt12__niter_baseIPcET_S1_;  lookup in file=/lib64/libm.so.6 [0]
10: symbol=_ZSt12__niter_baseIPcET_S1_;  lookup in file=/lib64/libgcc_s.so.1 [0]
10: symbol=_ZSt12__niter_baseIPcET_S1_;  lookup in file=/lib64/libc.so.6 [0]
10: symbol=_ZSt12__niter_baseIPcET_S1_;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
10: symbol=_ZSt12__niter_baseIPcET_S1_;  lookup in file=/var/runtime/liblambdaruntime.so [0]
10: binding file /var/runtime/liblambdaruntime.so [0] to /var/runtime/liblambdaruntime.so [0]: normal symbol `_ZSt12__niter_baseIPcET_S1_'
10: symbol=_ZSt12__niter_baseIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEET_N9__gnu_cxx17__normal_iteratorIS8_T0_EE;  lookup in file=dotnet [0]
10: symbol=_ZSt12__niter_baseIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEET_N9__gnu_cxx17__normal_iteratorIS8_T0_EE;  lookup in file=/lib64/libpthread.so.0 [0]
10: symbol=_ZSt12__niter_baseIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEET_N9__gnu_cxx17__normal_iteratorIS8_T0_EE;  lookup in file=/lib64/libdl.so.2 [0]
10: symbol=_ZSt12__niter_baseIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEET_N9__gnu_cxx17__normal_iteratorIS8_T0_EE;  lookup in file=/lib64/libstdc++.so.6 [0]
10: symbol=_ZSt12__niter_baseIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEET_N9__gnu_cxx17__normal_iteratorIS8_T0_EE;  lookup in file=/lib64/libm.so.6 [0]
10: symbol=_ZSt12__niter_baseIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEET_N9__gnu_cxx17__normal_iteratorIS8_T0_EE;  lookup in file=/lib64/libgcc_s.so.1 [0]
10: symbol=_ZSt12__niter_baseIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEET_N9__gnu_cxx17__normal_iteratorIS8_T0_EE;  lookup in file=/lib64/libc.so.6 [0]
10: symbol=_ZSt12__niter_baseIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEET_N9__gnu_cxx17__normal_iteratorIS8_T0_EE;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
10: symbol=_ZSt12__niter_baseIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEET_N9__gnu_cxx17__normal_iteratorIS8_T0_EE;  lookup in file=/var/runtime/liblambdaruntime.so [0]
10: binding file /var/runtime/liblambdaruntime.so [0] to /var/runtime/liblambdaruntime.so [0]: normal symbol `_ZSt12__niter_baseIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEET_N9__gnu_cxx17__normal_iteratorIS8_T0_EE'
10: symbol=_ZNK9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE4baseEv;  lookup in file=dotnet [0]
10: symbol=_ZNK9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE4baseEv;  lookup in file=/lib64/libpthread.so.0 [0]
10: symbol=_ZNK9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE4baseEv;  lookup in file=/lib64/libdl.so.2 [0]
10: symbol=_ZNK9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE4baseEv;  lookup in file=/lib64/libstdc++.so.6 [0]
10: symbol=_ZNK9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE4baseEv;  lookup in file=/lib64/libm.so.6 [0]
10: symbol=_ZNK9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE4baseEv;  lookup in file=/lib64/libgcc_s.so.1 [0]
10: symbol=_ZNK9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE4baseEv;  lookup in file=/lib64/libc.so.6 [0]
10: symbol=_ZNK9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE4baseEv;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
10: symbol=_ZNK9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE4baseEv;  lookup in file=/var/runtime/liblambdaruntime.so [0]
10: binding file /var/runtime/liblambdaruntime.so [0] to /var/runtime/liblambdaruntime.so [0]: normal symbol `_ZNK9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE4baseEv'
10: symbol=_ZSt13__copy_move_aILb0EPKcPcET1_T0_S4_S3_;  lookup in file=dotnet [0]
10: symbol=_ZSt13__copy_move_aILb0EPKcPcET1_T0_S4_S3_;  lookup in file=/lib64/libpthread.so.0 [0]
10: symbol=_ZSt13__copy_move_aILb0EPKcPcET1_T0_S4_S3_;  lookup in file=/lib64/libdl.so.2 [0]
10: symbol=_ZSt13__copy_move_aILb0EPKcPcET1_T0_S4_S3_;  lookup in file=/lib64/libstdc++.so.6 [0]
10: symbol=_ZSt13__copy_move_aILb0EPKcPcET1_T0_S4_S3_;  lookup in file=/lib64/libm.so.6 [0]
10: symbol=_ZSt13__copy_move_aILb0EPKcPcET1_T0_S4_S3_;  lookup in file=/lib64/libgcc_s.so.1 [0]
10: symbol=_ZSt13__copy_move_aILb0EPKcPcET1_T0_S4_S3_;  lookup in file=/lib64/libc.so.6 [0]
10: symbol=_ZSt13__copy_move_aILb0EPKcPcET1_T0_S4_S3_;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
10: symbol=_ZSt13__copy_move_aILb0EPKcPcET1_T0_S4_S3_;  lookup in file=/var/runtime/liblambdaruntime.so [0]
10: binding file /var/runtime/liblambdaruntime.so [0] to /var/runtime/liblambdaruntime.so [0]: normal symbol `_ZSt13__copy_move_aILb0EPKcPcET1_T0_S4_S3_'
10: symbol=_ZNSt11__copy_moveILb0ELb1ESt26random_access_iterator_tagE8__copy_mIcEEPT_PKS3_S6_S4_;  lookup in file=dotnet [0]
10: symbol=_ZNSt11__copy_moveILb0ELb1ESt26random_access_iterator_tagE8__copy_mIcEEPT_PKS3_S6_S4_;  lookup in file=/lib64/libpthread.so.0 [0]
10: symbol=_ZNSt11__copy_moveILb0ELb1ESt26random_access_iterator_tagE8__copy_mIcEEPT_PKS3_S6_S4_;  lookup in file=/lib64/libdl.so.2 [0]
10: symbol=_ZNSt11__copy_moveILb0ELb1ESt26random_access_iterator_tagE8__copy_mIcEEPT_PKS3_S6_S4_;  lookup in file=/lib64/libstdc++.so.6 [0]
10: symbol=_ZNSt11__copy_moveILb0ELb1ESt26random_access_iterator_tagE8__copy_mIcEEPT_PKS3_S6_S4_;  lookup in file=/lib64/libm.so.6 [0]
10: symbol=_ZNSt11__copy_moveILb0ELb1ESt26random_access_iterator_tagE8__copy_mIcEEPT_PKS3_S6_S4_;  lookup in file=/lib64/libgcc_s.so.1 [0]
10: symbol=_ZNSt11__copy_moveILb0ELb1ESt26random_access_iterator_tagE8__copy_mIcEEPT_PKS3_S6_S4_;  lookup in file=/lib64/libc.so.6 [0]
10: symbol=_ZNSt11__copy_moveILb0ELb1ESt26random_access_iterator_tagE8__copy_mIcEEPT_PKS3_S6_S4_;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
10: symbol=_ZNSt11__copy_moveILb0ELb1ESt26random_access_iterator_tagE8__copy_mIcEEPT_PKS3_S6_S4_;  lookup in file=/var/runtime/liblambdaruntime.so [0]
10: binding file /var/runtime/liblambdaruntime.so [0] to /var/runtime/liblambdaruntime.so [0]: normal symbol `_ZNSt11__copy_moveILb0ELb1ESt26random_access_iterator_tagE8__copy_mIcEEPT_PKS3_S6_S4_'
10: symbol=memmove;  lookup in file=dotnet [0]
10: symbol=memmove;  lookup in file=/lib64/libpthread.so.0 [0]
10: symbol=memmove;  lookup in file=/lib64/libdl.so.2 [0]
10: symbol=memmove;  lookup in file=/lib64/libstdc++.so.6 [0]
10: symbol=memmove;  lookup in file=/lib64/libm.so.6 [0]
10: symbol=memmove;  lookup in file=/lib64/libgcc_s.so.1 [0]
10: symbol=memmove;  lookup in file=/lib64/libc.so.6 [0]
10: binding file /var/runtime/liblambdaruntime.so [0] to /lib64/libc.so.6 [0]: normal symbol `memmove' [GLIBC_2.2.5]

Edit 2: I've narrowed it down to this line of code being the point at which the DllNotFoundException is thrown:

using (GurobiSolver solver = new GurobiSolver())

Answer 1

In the bin folder there are two files: libgurobi.so.9.5.0 and libgurobi95.so. The first is 41.5MB and I presume contains Gurobi runtimes. The second is just a text file containing the line ./libgurobi.so.9.5.0.

This gave me an "ah ha!" moment, and so I made a copy of libgurobi.so.9.5.0, renamed it to libgurobi95.so and replaced the original libgurobi95.so file with it in the zip file to upload to AWS.

And now it works. I don't fully understand why it works (why would the references be different when running on AWS than when running locally?). It may be something to do with the fact that when I run it locally, it's running in Windows, but AWS lambda functions run in Linux (Amazon Linux 2), but I don't really know.

It's aggravating that I have to go through that rigmarole every time I want to update the lambda function but at least it works.