ios - VNTrackRectangleRequest 内部错误

Question

我正在尝试让一个简单的矩形跟踪 Controller 运行，并且我可以让矩形检测正常运行，但跟踪请求总是以我无法找到的原因而告终。

有时跟踪请求会在失败前触发它的回调几次，有时它会在一次回调发生之前立即失败。我觉得这与我提交请求的方式有关，但我无法追根究底。

这是 View Controller 的代码

class TestController: UIViewController, AVCaptureVideoDataOutputSampleBufferDelegate {

    // Video capture
    private var videoSession = AVCaptureSession()
    private var videoLayer: AVCaptureVideoPreviewLayer!

    // Detection
    private var detectionRequest: VNDetectRectanglesRequest?
    private let sequenceHandler = VNSequenceRequestHandler()

    // Tracking
    private var trackingRequest: VNTrackRectangleRequest?
    private var shape: Detection?
    private var pixelBuffer: CVImageBuffer?




    // MARK: Setup
    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)

        startVideoFeed()
    }

    override func viewDidLayoutSubviews() {
        videoLayer.frame = view.layer.bounds
    }

    override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
        startDetectingRectangles()
    }



    private func startDetectingRectangles() {
        let request = VNDetectRectanglesRequest(completionHandler: didDetectRectangle)
        request.maximumObservations = 1
        request.minimumSize = 0.07
        request.minimumConfidence = 0.9
        request.minimumAspectRatio = 0.5
        request.maximumAspectRatio = 2
        request.quadratureTolerance = 10
        detectionRequest = request
    }

    private func didDetectRectangle(request: VNRequest, error: Error?) {

        // Fetch results of the correct type
        guard let observations = request.results, observations.count > 0 else { return }
        let results = observations.map { $0 as? VNRectangleObservation }

        for case let rectangle? in results {
            detectionRequest = nil

            let request = VNTrackRectangleRequest(rectangleObservation: rectangle, completionHandler: didTrackRectangle)
            trackingRequest = request
        }
    }


    private func didTrackRectangle(request: VNRequest, error: Error?) {

        // Fetch results of the correct type
        guard let observation = request.results?.first as? VNRectangleObservation else { return }

        // Create or update UI
    }




    // Start capturing video frames
    private func startVideoFeed() {

        // Session config
        videoSession.sessionPreset = .photo

        // Create device and input to device
        guard
            let captureDevice = AVCaptureDevice.default(for: .video),
            let deviceInput = try? AVCaptureDeviceInput(device: captureDevice)
            else {
                fatalError("Error setting up capture device.")
        }

        // Setup device output
        let deviceOutput = AVCaptureVideoDataOutput()
        deviceOutput.setSampleBufferDelegate(self, queue: DispatchQueue.global(qos: .default))

        // Set input and output
        videoSession.addInput(deviceInput)
        videoSession.addOutput(deviceOutput)

        // Setup video display layer
        videoLayer = AVCaptureVideoPreviewLayer(session: videoSession)
        videoLayer.frame = view.bounds
        videoLayer.videoGravity = .resizeAspectFill
        view.layer.addSublayer(videoLayer)

        videoSession.startRunning()
    }

    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {

        pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)

        guard let pixelBuffer = pixelBuffer else { return }
        do {

            var requests: [VNRequest] = []

            if let detectionRequest = detectionRequest {
                requests.append(detectionRequest) }
            if let trackingRequest = trackingRequest {
                requests.append(trackingRequest) }

            try sequenceHandler.perform(requests, on: pixelBuffer, orientation: .right)
        } catch {
            print(error)
        }
    }
}

打印错误:

Error Domain=com.apple.vis Code=9 "Internal error: Tracking of one of the corners failed, confidence = 0.000000; threshold = 0.650000" UserInfo={NSLocalizedDescription=Internal error: Tracking of one of the corners failed, confidence = 0.000000; threshold = 0.650000}

这是在跟踪请求第一次失败后每帧输出的，并且请求永远不会自行恢复。是 try sequenceHandler.perform(requests, on: pixelBuffer, orientation: .right) 行抛出错误(然后我打印)。

我查看了网络上散布的对象观察示例，它们似乎需要在每一帧发出新的跟踪请求来跟踪对象。我尝试在 didTrackRectangle 函数中创建一个新的 VNTrackRectangleRequest，这样每个帧都有一个新请求，但我遇到了同样的问题。

非常感谢与此相关的任何帮助。

Answer 1

我能够毫无问题地运行您的代码，至少在跟踪清晰可见的矩形时是这样。

当跟踪请求失败时，您可能应该创建一个新的 VNSequenceRequestHandler。似乎当跟踪请求失去对对象的跟踪时间太长时，它将无法恢复并重新开始跟踪它。这也意味着它们永远不会被释放，并且会不断抛出错误。如果您继续向请求处理程序添加新的跟踪请求，您很快就会超过同时跟踪请求的限制。

如果您能够预见到跟踪序列的结束，您可以设置 isLastFrame将您的跟踪请求的属性设置为 true，当当前帧完成处理时，它将被释放到可用跟踪器池中。