c# - AWS Toolkit for Visual Studio 错误？ AWS C# lambda 无法加载Kafka 的librdkafka 原生库？

Question

以下带有 Confluent.Kafka nuget 包的 .Net core 2.1 lambda 应用程序是在 Visual Studio 中使用适用于 Visual Studio 2017 的 AWS 工具包创建的。但是，以下代码

using (var producer = new Producer<Null, string>(config, null, new StringSerializer(Encoding.UTF8)))
{
    await producer.ProduceAsync("mytopic", null, msg);
}

在 CloudWatch 中出现以下错误

One or more errors occurred. (Failed to load the librdkafka native library.): AggregateExceptionat System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)at lambda_method(Closure , Stream , Stream , LambdaContextInternal )at Confluent.Kafka.Impl.LibRdKafka.Initialize(String userSpecifiedPath)at Confluent.Kafka.Producer..ctor(IEnumerable`1 config, Boolean manualPoll, Boolean disableDeliveryReports)at Confluent.Kafka.Producer`2..ctor(IEnumerable`1 config, ISerializer`1 keySerializer, ISerializer`1 valueSerializer, Boolean manualPoll, Boolean disableDeliveryReports)at StratsLambda.Function.FunctionHandler(S3Event evnt, ILambdaContext context) in C:\work\.....

Here is the AWS toolkit publishing messages:

Executing publish commandDeleted previous publish folder... invoking 'dotnet publish', working folder 'C:\work\....\bin\Release\netcoreapp2.1\publish'... publish: Microsoft (R) Build Engine version 15.7.179.6572 for .NET Core... publish: Copyright (C) Microsoft Corporation. All rights reserved.... publish:   Restore completed in 31.3 ms for C:\work\....\StratsLambda.csproj.... publish:   Restore completed in 62.87 ms for C:\work\....\StratsLambda.csproj.... publish:   StratsLambda -> C:\work\....\bin\Release\netcoreapp2.1\linux-x64\StratsLambda.dll... publish:   StratsLambda -> C:\work\....\bin\Release\netcoreapp2.1\publish\Zipping publish folder C:\work\....\bin\Release\netcoreapp2.1\publish to C:\work\....\bin\Release\netcoreapp2.1\StratsLambda.zip... zipping: Amazon.Lambda.Core.dll... zipping: Amazon.Lambda.S3Events.dll... zipping: Amazon.Lambda.Serialization.Json.dll... zipping: AWSSDK.CloudWatchLogs.dll... zipping: AWSSDK.Core.dll... zipping: AWSSDK.S3.dll... zipping: Confluent.Kafka.dll... zipping: debian9-librdkafka.so... zipping: librdkafka.so... zipping: Newtonsoft.Json.dll... zipping: Serilog.dll... zipping: Serilog.Sinks.AwsCloudWatch.dll... zipping: Serilog.Sinks.PeriodicBatching.dll... zipping: StratsLambda.deps.json... zipping: StratsLambda.dll... zipping: StratsLambda.pdb... zipping: StratsLambda.runtimeconfig.json... zipping: System.Runtime.CompilerServices.Unsafe.dllUpdating code for existing function savstratsConfig settings saved to C:\work\....\aws-lambda-tools-defaults.json

I created a unit test function to test the Kafka producing and it can send message to my Kafka successfully. And the folder ...\bin\Debug\netcoreapp2.1 has only the following files.

Name                                Length----                                ------StratsLambda.deps.json               58489StratsLambda.dll                     12800StratsLambda.pdb                      1908StratsLambda.runtimeconfig.dev.json    238StratsLambda.runtimeconfig.json        154

---

Update:The file StratsLambda.deps.json has the following section,

  "librdkafka.redist/0.11.5": {
    "native": {
      "runtimes/linux-x64/native/centos7-librdkafka.so": {
        "fileVersion": "0.0.0.0"
      },
      "runtimes/linux-x64/native/debian9-librdkafka.so": {
        "fileVersion": "0.0.0.0"
      },
      "runtimes/linux-x64/native/librdkafka.so": {
        "fileVersion": "0.0.0.0"
      }
    }
  },

我还尝试在 zip 文件中创建文件夹 runtimes/linux-x64/native 和电影文件。

但是，还是报错

An assembly specified in the application dependencies manifest (MyApp.deps.json) was not found:

package: 'librdkafka.redist', version: '0.11.5'

path: 'runtimes/linux-x64/native/centos7-librdkafka.so'

START RequestId: 73e39d0b-8ec8-11e8-8597-.... Version: $LATEST

Failed to execute the Lambda function. The dotnet CLI failed to start with the provided deployment package. Please check CloudWatch logs for this Lambda function to get detailed information about this failure.: LambdaException

Answer 1

正如它所提到的，因为这些库只是运行时的一部分，并且需要特定的运行时才能在主机上可用。

我建议不要在您的源代码中包含运行时目录，它可能有效但看起来很难看。

我使用了以下解决方案。

已添加 <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>项目文件中的标记。下面的示例；

或者，您也可以添加 -p:GenerateRuntimeConfigurationFiles=true作为 dotnet publish 的参数在您的构建服务器上执行命令。

上面应该输出runtime你的目录bin (或输出)目录。如果您使用 nuget 构建包，那么您还需要确保您的 *.nuspec (如果这是用于打包的)还包括 nuget 包中的目录。下面的示例包括包中的所有文件和所有目录。