PHP fatal error : Allowed memory size of 268435456 bytes exhausted on NormalizerFormatter. php 第 62 行

Question

我正在尝试通过 doctrine2 在 mysql 中插入大约 20000 条记录

    foreach ($getIds as $id){// 20000 Ids
            $twNode = new TwNode();
            $twNode->setTwId($id);
            $twNode->addAcFriend($this->twAccount);
            $this->twAccount->addFollower($twNode);
            $this->em->persist($twNode);
            $this->em->persist($this->twAccount);
            $this->em->flush();
            $this->output->write('W');
            $followersWrite++;
        }
    }

但是发生了这个错误，我怎样才能避免这个 Allowed memory 错误？

PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes) in /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php on line 62
PHP Stack trace:
PHP   1. {main}() /Users/whitebear/httproot/myapp/app/console:0
PHP   2. Symfony\Component\Console\Application->run() /Users/whitebear/httproot/myapp/app/console:22
PHP   3. Symfony\Bundle\FrameworkBundle\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:106
PHP   4. Symfony\Component\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:83
PHP   5. Symfony\Component\Console\Command\Command->run() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:200
PHP   6. Acme\TopBundle\Command\twCrawlerCommand->execute() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:242
PHP   7. Acme\TopBundle\Command\twCrawlerCommand->initialRun() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:430
PHP   8. EntityManager52ee6dcbdb2bb_546a8d27f194334ee012bfe64f629947b07e4919\__CG__\Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:284
PHP   9. Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/app/cache/dev/jms_diextra/doctrine/EntityManager_52ee6dcbdb2bb.php:402
PHP  10. Doctrine\ORM\UnitOfWork->commit() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:389
PHP  11. Doctrine\ORM\UnitOfWork->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:335
PHP  12. Doctrine\ORM\Persisters\BasicEntityPersister->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:952
PHP  13. Doctrine\DBAL\Statement->execute() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/BasicEntityPersister.php:279
PHP  14. Doctrine\DBAL\Logging\LoggerChain->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php:161
PHP  15. Symfony\Bridge\Doctrine\Logger\DbalLogger->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Logging/LoggerChain.php:54
PHP  16. Symfony\Bridge\Doctrine\Logger\DbalLogger->log() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:80
PHP  17. Monolog\Logger->debug() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:102
PHP  18. Monolog\Logger->addRecord() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:423
PHP  19. Monolog\Handler\AbstractProcessingHandler->handle() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:244
PHP  20. Monolog\Formatter\LineFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php:35
PHP  21. Monolog\Formatter\NormalizerFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:43
PHP  22. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:40
PHP  23. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88
PHP  24. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:70
PHP  25. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes) in /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php on line 62

Call Stack:
    0.0002     229968   1. {main}() /Users/whitebear/httproot/myapp/app/console:0
    0.0126    2063376   2. Symfony\Component\Console\Application->run() /Users/whitebear/httproot/myapp/app/console:22
    0.0144    2256968   3. Symfony\Bundle\FrameworkBundle\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:106
    0.1503   11751600   4. Symfony\Component\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:83
    0.1518   11753384   5. Symfony\Component\Console\Command\Command->run() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:200
    0.1521   11758040   6. Acme\TopBundle\Command\twCrawlerCommand->execute() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:242
    0.2409   20673112   7. Acme\TopBundle\Command\twCrawlerCommand->initialRun() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:430
  101.5883  235069104   8. EntityManager52ee6dcbdb2bb_546a8d27f194334ee012bfe64f629947b07e4919\__CG__\Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:284
  101.5883  235069200   9. Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/app/cache/dev/jms_diextra/doctrine/EntityManager_52ee6dcbdb2bb.php:402
  101.5883  235069280  10. Doctrine\ORM\UnitOfWork->commit() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:389
  107.0744  259211408  11. Doctrine\ORM\UnitOfWork->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:335
  107.1128  259277872  12. Doctrine\ORM\Persisters\BasicEntityPersister->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:952
  109.1604  267877360  13. Doctrine\DBAL\Statement->execute() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/BasicEntityPersister.php:279
  109.1604  267877408  14. Doctrine\DBAL\Logging\LoggerChain->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php:161
  109.1604  267877408  15. Symfony\Bridge\Doctrine\Logger\DbalLogger->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Logging/LoggerChain.php:54
  109.1604  267877592  16. Symfony\Bridge\Doctrine\Logger\DbalLogger->log() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:80
  109.1604  267877592  17. Monolog\Logger->debug() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:102
  109.1604  267877640  18. Monolog\Logger->addRecord() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:423
  109.1604  267878968  19. Monolog\Handler\AbstractProcessingHandler->handle() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:244
  109.1604  267878968  20. Monolog\Formatter\LineFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php:35
  109.1604  267878968  21. Monolog\Formatter\NormalizerFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:43
  109.1604  267878968  22. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:40
  109.1604  267878968  23. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88
  109.1605  267881872  24. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:70
  109.1605  267881872  25. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88

感谢您的评论，我了解到它与 $this->em 消耗的内存有关

因此，我将代码每 5000 次取消设置 $this->em；

if ($k > 1000){
    $this->em->flush();
    unset($this->em);
    $this->em = $this->getContainer()->get('doctrine')->getManager();
}

但仍然是徒劳的，我想我应该取消 $this->getContainer()->get('doctrine')->getManager(); 的起源;

这可能吗？或者我是对的？

Answer 1

您可能想阅读 Doctrine ORM Batch Processing .他们正在使用这种模式:

// $em instanceof EntityManager
$batchSize = 20;
for ($i=1; $i<=10000; ++$i) {
     $obj = new MyEntity;
     $obj->setFoo('...');
     // ... set more data
     $em->persist($obj);
     if (($i % $batchSize) == 0) {
         $em->flush();
         $em->clear();
    }
}
$em->flush();

据我了解 - 在坚持之后，学说将所有映射对象保存在标识映射中的内存中。因此，随着每个新对象的出现，内存需求一直在增长。如果您执行 clear()，它将分离所有托管实体，如果您没有对它们的引用，它们将被垃圾收集。

注意:它会分离您正在使用的所有其他实体。或者你可以只分离那些插入的对象 $em->detach($twNode);//与 Doctrine 分离，以便立即进行 GC

编辑:在 for 循环之后添加 flush() 以确保保存所有实体，即使实体计数不能被批量大小整除，如评论中所建议的