I have a course scheduling application based on OptaPlanner 8.22.1.Final.

我有一个基于OptaPlanner 8.22.1的排课应用程序。

After upgrading to Timefold 1.1.0, the execution time for the performance test cases increases about 100%. The application code is the same other than changes to point to the Timefold library. The JDK goes up from 11 to 17.

升级到TimeFold1.1.0后，性能测试用例的执行时间增加了大约100%。应用程序代码相同，只是更改为指向TimeFold库。JDK指数从11上升到17。

Here are some details about the tests.

以下是有关测试的一些细节。

Test case 1

测试用例1

Before

在此之前

2023-09-10 13:02:17,390 INFO Solving started: time spent (5), best score (-730init/0hard/0medium/0soft), environment mode (REPRODUCIBLE), move thread count (4), random (JDK with seed 0).
2023-09-10 13:02:17,902 INFO Construction Heuristic phase (0) ended: time spent (517), best score (-40hard/-2265medium/0soft), score calculation speed (32158/sec), step total (365).
2023-09-10 13:03:31,283 INFO Local Search phase (1) ended: time spent (73898), best score (0hard/0medium/0soft), score calculation speed (24932/sec), step total (28873).
2023-09-10 13:03:31,283 INFO Solving ended: time spent (73898), best score (0hard/0medium/0soft), score calculation speed (24979/sec), phase total (2), environment mode (REPRODUCIBLE), move thread count (4).

After

之后

2023-09-10 15:38:24,216 INFO Solving started: time spent (11), best score (-730init/0hard/0medium/0soft), environment mode (REPRODUCIBLE), move thread count (4), random (JDK with seed 0).
2023-09-10 15:38:24,590 INFO Construction Heuristic phase (0) ended: time spent (385), best score (-40hard/-2265medium/0soft), score calculation speed (87852/sec), step total (365).
2023-09-10 15:42:03,882 INFO Local Search phase (1) ended: time spent (219677), best score (0hard/-10medium/0soft), score calculation speed (35648/sec), step total (31041).
2023-09-10 15:42:03,883 INFO Solving ended: time spent (219677), best score (0hard/-10medium/0soft), score calculation speed (35734/sec), phase total (2), environment mode (REPRODUCIBLE), move thread count (4).

Test case 2:

测试用例2：

Before

在此之前

2023-09-10 13:03:32,508 INFO Solving started: time spent (16), best score (-3796init/0hard/0medium/0soft), environment mode (REPRODUCIBLE), move thread count (4), random (JDK with seed 13).
2023-09-10 13:03:34,728 INFO Construction Heuristic phase (0) ended: time spent (2236), best score (-10hard/-6460medium/0soft), score calculation speed (40084/sec), step total (1898).
2023-09-10 13:08:37,166 INFO Local Search phase (1) ended: time spent (304674), best score (0hard/0medium/0soft), score calculation speed (13550/sec), step total (83120).
2023-09-10 13:08:37,167 INFO Solving ended: time spent (304675), best score (0hard/0medium/0soft), score calculation speed (13742/sec), phase total (2), environment mode (REPRODUCIBLE), move thread count (4).

After

之后

2023-09-10 15:42:04,616 INFO Solving started: time spent (32), best score (-3796init/0hard/0medium/0soft), environment mode (REPRODUCIBLE), move thread count (4), random (JDK with seed 13).
2023-09-10 15:42:07,385 INFO Construction Heuristic phase (0) ended: time spent (2801), best score (-10hard/-6460medium/0soft), score calculation speed (64265/sec), step total (1898).
2023-09-10 15:52:28,340 INFO Local Search phase (1) ended: time spent (623756), best score (0hard/0medium/0soft), score calculation speed (12726/sec), step total (82742).
2023-09-10 15:52:28,341 INFO Solving ended: time spent (623757), best score (0hard/0medium/0soft), score calculation speed (12954/sec), phase total (2), environment mode (REPRODUCIBLE), move thread count (4).

I also tried OptaPlanner 8.37.0.Final and a few other version after the introduction of Bavet. They all caused performance degradation.

在引入Bavet之后，我也尝试了OptaPlanner 8.37.0Final和其他几个版本。它们都会导致性能下降。

What changes do I need to make to make the application run faster? I was expect the application to run slightly faster.

我需要进行哪些更改才能使应用程序运行得更快？我预计应用程序的运行速度会稍微快一些。

We believe the original code might be overfitting and the change in JDK version exposed that. If that's true, the change in OptaPlanner/Timefold version isn't relevant. In fact, the upgrade will probably improve your production quality (= paradox).

我们认为原始代码可能过多，而JDK版本的更改暴露了这一点。如果这是真的，那么OptaPlanner/Timeold版本的变化就无关紧要了。事实上，升级可能会提高你的产品质量。

A benchmark on the same JDK version should (dis)prove that.

同一个JDK版本上的基准测试应该(不)证明这一点。

Motivation

动机

1. The testcases use a different random seed according to the log:

• Testcase 1 (before+after): Solving started: ... random (... seed 0)


• Testcase 2 (before+after): Solving started: ... random (... seed 13)

That is very unusual. It's a sign of overfitting. Overfitting gives better results during testing, but equal or worse results in production.
I suspect optaplanner-benchmarker has been (mis)used to find the "best random seed" for each dataset. That makes it vulnerable to any changes in the random implementation.

这是非常不寻常的。这是一种过度适应的迹象。过拟合会在测试过程中提供更好的结果，但在生产中会产生相同或更差的结果。我怀疑optaplanner基准已经被(错误地)用来为每个数据集寻找“最佳随机种子”。这使得它很容易受到随机实现中的任何更改的影响。

2. JDK 17 changed the random implementation.

   
   



3. The time spent more than doubled, but the score calculation speed and LS steps remained the same. That means the number of moves per step more than doubled. It just generated more unlucky moves.

   
   

From looking at the test logs, it seems to me that you're comparing apples to oranges. Specifically:

从测试日志来看，在我看来，你是在拿苹果和橙子做比较。具体地说，就是：

Test case 1:

测试用例1：

• "before" ran for ~ 1 minute 15 seconds


• "after" ran for ~ 3 minutes 30 seconds.


• the result "after" was significantly faster than "before". (~ +45 %)

Test case 2:

测试用例2：

• "before" ran for ~ 5 minutes.


• "after" ran for ~ 10 minutes.


• the result "after" was only worse by about 6 %, which can be easily explained by natural variance in JVM performance from one run to another.

Considering that the input conditions (specifically solving time or target score) are not the same and that this is not a scientific benchmark, I would be cautious in drawing any conclusions from it.

考虑到输入条件(特别是求解时间或目标分数)不同，而且这不是一个科学的基准，我将谨慎地从中得出任何结论。

Here are a few guidelines to improve the reliability of your benchmarks:

以下是提高基准可靠性的一些指导原则：

• Set a shared termination condition. If you're measuring performance, time-based terminations aren't ideal. Maybe a score-based or a step count-based termination.


• Run each experiment multiple times, average the results. 10 iterations each should suffice.


• Try to eliminate variables. If you say you've reproduced this with OptaPlanner 8, don't benchmark Timefold (yet). Instead, try to reproduce it without a switch to JDK 17. That way, we'll know if it's JDK-related or not.

As a final note, it is not the first time that I'm hearing that something happened some time after OptaPlanner 8.22 which caused it to go significantly slower in some cases. Unfortunately, no one has yet provided code that would show the slowdown. You can be the first.

作为最后的提示，这不是我第一次听说OptaPlanner 8.22之后的一段时间发生了什么事情，在某些情况下，它的运行速度明显变慢了。不幸的是，目前还没有人提供显示经济放缓的代码。你可以是第一个。