gpt4 book ai didi

azure - C#Map Reduce失败,出现 “{”响应状态代码并不表示成功:403(禁止)。“}有时是401:需要凭据

转载 作者:行者123 更新时间:2023-12-02 21:14:12 24 4
gpt4 key购买 nike

System.AggregateException中发生了mscorlib.dll类型的未处理异常

内部异常(exception):{"Response status code does not indicate success: 403 (Forbidden)."}有时得到:{"Response status code does not indicate success: 401 (Credentials required)."}
所有登录名均正确。
Hadoop.Connect()正确连接。

堆栈跟踪:

at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at System.Threading.Tasks.Task.Wait() at Microsoft.Hadoop.WebClient.WebHCatClient.WebHcatMapReduceStreamingExecutor.Execute(Boolean throwOnError) at Microsoft.Hadoop.MapReduce.Execution.Hadoop.StreamingJobExecutorBase.ExecuteCore(Type mapper, Type reducer, Type combiner, HadoopJobConfiguration config)
at Microsoft.Hadoop.MapReduce.Execution.Hadoop.StreamingJobExecutorBase.Execute(Type mapperType, Type reducerType, Type combinerType, HadoopJobConfiguration config) at Microsoft.Hadoop.MapReduce.Execution.Hadoop.StreamingJobExecutorBase.Execute[TMapper,TReducer](HadoopJobConfiguration config) at Pwc.Demo.HDPClient.Program.Main(String[] args) in C:\ASC Project Info\TalentExchange\Demo project\Pwc.Demo.HDPClient\Pwc.Demo.HDPClient\Program.cs:line 49 at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args) at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args) at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart()


class Program
{
static void Main(string[] args)
{
HadoopJobConfiguration myConfig = new HadoopJobConfiguration();

myConfig.InputPath = "asv://hdsto-something-.net/data/in/reviews.txt";
myConfig.OutputFolder = "asv://hd-something-.blob.core.windows.net/data/out/";
Environment.SetEnvironmentVariable("HADOOP_HOME", @"C:\apps\dist\hadoop-2.7.1.2.3.3.1-25\bin");
Environment.SetEnvironmentVariable("JAVA_HOME", @"C:\Program Files\Java\jre1.8.0_101\bin\javaw.exe");

Uri azureCluster = new Uri("https://somename--Demo.azurehdinsight.net");
string clusterUserName = "***";
string clusterPassword = "****";

// This is the name of the account under which Hadoop will execute jobs.
// Normally this is just "Hadoop".
string hadoopUserName = "Hadoop";

// Azure Storage Information.
string azureStorageAccount = "somestore.blob.core.windows.net";
string azureStorageKey = "***==";
string azureStorageContainer = "**";
bool createContinerIfNotExist = false;

IHadoop myCluster = Hadoop.Connect(azureCluster,
clusterUserName,
hadoopUserName,
clusterPassword,
azureStorageAccount,
azureStorageKey,
azureStorageContainer,
createContinerIfNotExist);

//execute mapreduce job

MapReduceResult jobResult =

myCluster.MapReduceJob.Execute<MySimpleMapper, MySimpleReducer>(myConfig);

int exitCode = jobResult.Info.ExitCode;
string exitStatus = "Failure";
if (exitCode == 0) exitStatus = "Success";
exitStatus = exitCode + " (" + exitStatus + ")";
Console.WriteLine();
Console.Write("Exit Code = " + exitStatus);
Console.Read();
}
}

public class MySimpleMapper : MapperBase
{
public override void Map(string inputLine, MapperContext context)
{
int value = int.Parse(inputLine);
string key = (value % 2 == 0) ? "even" : "odd";
context.EmitKeyValue(key, value.ToString());
}
}

public class MySimpleReducer : ReducerCombinerBase
{
public override void Reduce(string key, IEnumerable<string> values, ReducerCombinerContext context)
{
//initialize counters
int myCount = 0;
int mySum = 0;
//count and sum incoming values
foreach (string value in values)
{
mySum += int.Parse(value);
myCount++;
}
context.EmitKeyValue(key, myCount + "t" + mySum);

}
}

错误快照:
enter image description here

最佳答案

您的输入路径不应是特定文件,而应是目录。该目录应仅包含文件,而不应包含文件夹。

myConfig.InputPath = "asv://hdsto-something-.net/data/in/";

关于azure - C#Map Reduce失败,出现 “{”响应状态代码并不表示成功:403(禁止)。“}有时是401:需要凭据,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/39286123/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com