Difference between timestamp file and Last Index update time on web UI

[oacnhpkqxjhufwumkkqnshvlmheokqv]

Describe the bug
We have a Nagios monitor configured to check the last modified time of the timestamp file, but, that timestamp was being updated, when the "Last Index update" time on the bottom of the OpenGrok page showed a timestamp a week old.

Once we rebooted the VM and let it run for a few hours, then the timestamps matched.

It's a separate issue that our indexer hung up, this concern is around the timestamp file being updated periodically, when the UI shows a different time. Where does the web UI get its timestamp from, so I can change the Nagios monitor to check there?

Or would this be related to timestamps from indexing different projects #1670 ?

OpenGrok v1.3.8
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b01-1~deb9u1-b01)
OpenJDK 64-Bit Server VM (build 25.212-b01, mixed mode)
OS: Debian GNU/Linux 9 (stretch)

Server version: Apache Tomcat/9.0.20
OS Version: 3.10.0-1127.el7.x86_64
JVM Version: 1.8.0_212-8u212-b01-1~deb9u1-b01

To Reproduce
Steps to reproduce the behavior:

• Not sure what's leading to this

Expected behavior
A clear and concise description of what you expected to happen.

• Timestamp on /opengrok/data/timestamp file to match the timestamp shown in the web UI.

Screenshots
If applicable, add screenshots to help explain your problem.

OpenGrok web UI shows Last Index update Nov 11

Nagios monitor flapping. Monitor is configured to warn if the time stamp is > 4 hours old, and alert if > 6 hours old. We have the indexer running every 2 hours.

Additional context
Add any other context about the problem here.

[vladak]

W.r.t. #1670 - this only matters if per project indexing is used in which case the timestamp is not touched at all. Do you index each project separately ? How do you run the indexer ?

[vladak]

I have a sort of a deja-vu - thinking that the timestamp file mechanics was already explained at length somewhere however I cannot find the issue. So with the risk of duplication:

The timestamp of the last index is the last modified time of the timestamp file that resides directly under data root. The IndexTimestamp class manages it. It has 3 methods:

• getDateForLastIndexRun() - returns cached time stamp value (to avoid unnecessary I/O). This is called from PageConfig to display the timestamp value at the bottom of the page in the web application.
• refreshDateForLastIndexRun() - resets the caching so that fresh time stamp is returned after the next call to getDateForLastIndexRun() - this is called from RuntimeEnvironment#applyConfig() which is called when the configuration is refreshed. The indexer calls the RESTful API endpoint to refresh the configuration at the end of the indexing when run with the -U option.
• stamp() - touch the timestamp file. When the indexer is run, this is called at the very end of IndexDatabase#update().

So, for the timestamp value to change the Indexer must successfully complete the index database update. For the value to be reflected in the UI the new configuration must be successfully sent and processed by the web application. Therefore, if you have problems with the Indexer hanging then I'd say this can be closely related.

[oacnhpkqxjhufwumkkqnshvlmheokqv]

We're running the indexer with this: https://github.com/oracle/opengrok/blob/5b7cea89ad9c1b12ee0f49c4467801d98f50d188/docker/index.sh

How do I find if we're indexing each project separately?

Thanks!

[vladak]

We're running the indexer with this: https://github.com/oracle/opengrok/blob/5b7cea89ad9c1b12ee0f49c4467801d98f50d188/docker/index.sh

That's definitely running the index wholesale (at least for the time being until #3221 is addressed), i.e. for all projects at once.

How do I find if we're indexing each project separately?

It would have to be using the work style described on https://github.com/oracle/opengrok/wiki/Per-project-management-and-workflow which the current Docker image does not, so don't worry about this.