[scalability] SVN update takes hours if "Synchronize" view is opened
| Bugzilla Link | 504058 |
| Status | NEW |
| Importance | P3 normal |
| Reported | Oct 05, 2016 07:57 EDT |
| Modified | Oct 31, 2016 15:23 EDT |
| Version | 3.0.2 |
| Depends on | 505611 |
| See also | 504568, 505360 |
| Reporter | Andrey Loskutov |
Description
We have observed that "Update To" invoked from the History view can take literally hours if we had a Synchronize view opened and populated before.
We had a few commits in the SVN with changes on more then 1000 files each. Actual SVN update to the commit containing 10000 changed files takes ~3 minutes on command line, but the task triggered from Eclipse spins three jobs "Update To Operation", "Update SVN Cache", "Updating Synchronize view for SVN Workspace" which together take more then a hour, keeping up to two CPU cores 100% busy.
Closing the "Synchronize" view before update fixes the "hours" update to few minutes, still longer as command line, but at least not a hour anymore.
The problem can be observed by using default Subversive settings. I've tried to play with different Subversive settings but so far found nothing really helpful.
What also interesting is that the progress view shows "Updating" for resources which should have never been managed by SVN (they are both derived AND excluded via Team ignore hints).
I've debugged this a lot last few days, without real understanding of all this "team synchronize" design, which seem to be a huge over-engineered framework written at time of CVS and probably designed around CVS deficiencies. I cannot imagine this can be made reasonably fast at any time. Additionally, it is almost impossible to debug because of multitudes of different jobs and listeners sending events to other listeners which spin their own jobs to send even more events again to some listeners...
The main issue here is that due the "event per change" design, and LOT of redundant events sent/proceed by the team framework for each single change, Eclipse simply overloaded with that event handling. Funny thing to observe that JVM uses 100% of the CPU doing literally nothing for a hour after all SVN operations were finished, while dispatching gazillions of events in org.eclipse.team.internal.core.BackgroundEventHandler.processEvents(IProgressMonitor). BTW this specific observation about BackgroundEventHandler is related to the "Updating Synchronize view for SVN Workspace" job.
Unfortunately I do not have a public SVN server for reproducing this, but I will try to find one.
The key points for reproducing the problem:
- huge project (we have ~15000 versioned files in ~300 folders, and ~10000 derived files in ~300 derived folders)
- big remote commit containing > 10000 changed files
- Opened "Synchronize" view
- SVN update to that huge commit
=> leads to ~2 hours overall update time in Eclipse, compared with 3 minutes on command line.
This happens with our (3.0.2) and latest available (4.x) Subversive versions, on both Eclipse 3.8.2 and 4.6.1, with SVN connector 5.0.1 and SVNKit 1.7. (SVN/1.7.11 SVNKit/1.7.14 (http://svnkit.com/) r10415_v20150322_2135).
I think I have some kind of fix for this (bigger change), but I have to cleanup the code and cherry pick it to 4.x subversive branch before I can post it.
Activity
By Andrey Loskutov
I've mirrored the subversive repository to github and pushed my changes to the
https://github.com/iloveeclipse/subversive/tree/iloveeclipse branch.To see what was changed compared to master, see this pull request below:
https://github.com/iloveeclipse/subversive/pull/1The 2 most important commits are:
https://github.com/iloveeclipse/subversive/pull/1/commits/200e28fb8f8904f6de8bdfb4a9e5c612da7fcc51 which discards LOT of identical resource refresh tasks etc.
https://github.com/iloveeclipse/subversive/pull/1/commits/0b1a1c244c5a835034572a757361e58000cac43a which doesn't add ignored folders to the resource events
With that all I'm down from ~1-2 hours for an update with a commit containing 10000 files to ~5-10 minutes.
Please note that I've did the changes and tested them on 3.0.2 stream, and I need some time to re-build my environment to 4.0.x verify if the rebased change is OK on 4.0.2. However I don't think that there should be any significant differences compared with 3.0.2, because 4.0.2 showed same (abysmal) performance in our case.
By Andrey Loskutov
(In reply to Andrey Loskutov from comment #1)
Please note that I've did the changes and tested them on 3.0.2 stream, and I need some time to re-build my environment to 4.0.x verify if the rebased change is OK on 4.0.2. However I don't think that there should be any significant differences compared with 3.0.2, because 4.0.2 showed same (abysmal) performance in our case.
I've just switched over to 4.0 head and patches are working there as expected.
First of all, thank you for your help! And very much at that! :)
I applied most of the changes, what was excluded at the moment is "performance improvements for SVNLightweightDecorator ". The reason is as such: ResourceStatesChangedEvent.getResourcesRecursivelly() returns the exact set of resources to be processed, since it includes not just a set of roots, but a depth too. So, the actual resource set (roots+depth) is created elsewhere and if we receive an event that tells us to handle it recursively, then so be it. It seems the function's name is confusing, it would be better to call it something like "gatherResources()" or something in lines. Unfortunately it will cause an API level change, so for the time being it will stay like it is now.
Then there is a change that causes me some great doubts - "performance improvements for ResourceStatesChangedEvent". The reason is that I can manually add an ignored resource to the source control, but SVNUtility.isIgnored(resource) will still tell it is ignored, since it is just a hint. In the end event regarding the changes to those resources will be lost. Though, I do understand the reason you wanted to exclude these nodes. I commented the change currently, since I want to know your opinion regarding the matter first.
By the way, the same is problem is related to the "Bug 505360 - [scalability] Resource listener listens to changes on ignored resources" too.As for the "performance improvements for AbstractSVNSubscriber " - I applied it wholly, but on top of it I removed some code that seemed to be a legacy one and made some fixes in other parts. Both changes should further reduce performance overhead.
By Andrey Loskutov
Regarding listening to ignored files or react on changes for such files: last time I've tried to play with the plugin I saw there was some preference like "try to track ignored or derived files" (I can't remember the full option name). I have not found where or how it was used but I think this is something we should apply here too. By default derived (and also ignored) files should not be consodered as important. If someone has use cases where derived (or ignored) files should be managed by svn, the preference can be used or ignore patterns adopted. But it cannot be that such use cases are supported "by default", leaving the 95% of the "normal" users with incredible performance issues.
Regarding the rest: I have no access to my setup with SVN now, but may be in a week or so I could try the latest HEAD state with our SVN killer commits, and I will report if it fully fixes our performance issues or if there is something missing. Thanks for looking into this anyway.
BTW, if you have a good example of a public SVN server which can be used to test performance relates issies like this one, it would be nice if you would share the url here.(In reply to Andrey Loskutov from comment #4)
Regarding listening to ignored files or react on changes for such files: last time I've tried to play with the plugin I saw there was some preference like "try to track ignored or derived files" (I can't remember the full option name). I have not found where or how it was used but I think this is something we should apply here too. By default derived (and also ignored) files should not be consodered as important. If someone has use cases where derived (or ignored) files should be managed by svn, the preference can be used or ignore patterns adopted. But it cannot be that such use cases are supported "by default", leaving the 95% of the "normal" users with incredible performance issues.
Derived resources are handled bye this function exactly (SVNUtility.isIgnored()), so they'll be processed as you say. Still there're Team services hints included (Team.isIgnoredHint(resource)) too. That being said, I think you're right, it is better reduce performance overhead, than to handle such a rare cases.
Regarding the rest: I have no access to my setup with SVN now, but may be in a week or so I could try the latest HEAD state with our SVN killer commits, and I will report if it fully fixes our performance issues or if there is something missing. Thanks for looking into this anyway. BTW, if you have a good example of a public SVN server which can be used to test performance relates issies like this one, it would be nice if you would share the url here.
Unfortunately I've no idea. There're servers with big projects (like Apache Foundation's one), but it's not like they're test servers and could be used to test performance while performing commits etc.
