Daily Log for #alfresco IRC Channel

Alfresco discussion and collaboration. Stick around a few hours after asking a question.

Official support for Enterprise subscribers: support.alfresco.com.

Joining the Channel:

Join in the conversation by getting an IRC client and connecting to #alfresco at Freenode. Our you can use the IRC web chat.

More information about the channel is in the wiki.

Getting Help

More help is available in this list of resources.

Daily Log for #alfresco

2017-02-12 16:33:00 GMT <yreg> hey AFaust !

2017-02-12 16:33:10 GMT <AFaust> hey Younes

2017-02-12 16:33:42 GMT <yreg> I have been having a hard time configuring alfresco under windows !

2017-02-12 16:34:01 GMT <yreg> by configuring I meant tuning jvm params

2017-02-12 16:34:27 GMT <yreg> I have this setup with separate servers for Alfresco and Solr

2017-02-12 16:34:36 GMT <yreg> both windows

2017-02-12 16:34:57 GMT <yreg> Alfresco VM has 18G of ram and 4 cpu

2017-02-12 16:35:18 GMT <yreg> solr one has 48 G of ram and 8 cpu

2017-02-12 16:35:27 GMT <AFaust> Ok - so not locally but a "real" server

2017-02-12 16:36:00 GMT <yreg> Virtual machines on client's insfrastructure

2017-02-12 16:37:24 GMT <AFaust> And what specifically are the problems?

2017-02-12 16:37:44 GMT <yreg> I gave alfresco 14 G of ram and left 4 for the OS (we do connect from time to time to these servers using remote desktop to do all sorts of things)

2017-02-12 16:38:52 GMT <yreg> and gave solr 31 G (as I figured that any value >= 32G of RAM and <= 48G would have less performance than 31G

2017-02-12 16:39:27 GMT <yreg> well I am using the recommended GC config from alfresco and that seems to be not very optimal

2017-02-12 16:39:54 GMT <yreg> solr tomcat crashes from time to time, due to eden space being full

2017-02-12 16:40:43 GMT <yreg> that's one issue

2017-02-12 16:41:13 GMT <yreg> the other one is that I am now doing a full reindex

2017-02-12 16:41:56 GMT <AFaust> hehe - the recommended GC config from Alfresco is crap in my opinion

2017-02-12 16:41:57 GMT <yreg> and when I restart both servers the reindex process seems fast at first then starts to slow down

2017-02-12 16:42:06 GMT <AFaust> At least when it comes to Repository / Share tiers.

2017-02-12 16:42:36 GMT <AFaust> SOLR is another matter - the default isn't great for that but an alternative is not as clear cut

2017-02-12 16:42:38 GMT <yreg> tried to restart multiple times and I can confirm that whenever I restart the indexing is very fast at first and it gets slow after a while

2017-02-12 16:43:28 GMT <AFaust> Why do you figure that any value >= 326 and <= 48G would have less performance than 31 G?

2017-02-12 16:43:45 GMT <yreg> and I am thinking this should be related to the Garbage collection policy and jvm configuration

2017-02-12 16:44:20 GMT <yreg> AFaust, https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/

2017-02-12 16:44:22 GMT <alfbot> Title: Why 35GB Heap is Less Than 32GB - Java JVM Memory Oddities - codecentric AG Blog : codecentric AG Blog (at blog.codecentric.de)

2017-02-12 16:44:38 GMT <AFaust> The problem with large heap sizes is that CMS GC is very rigid/unflexible and a) effectively wastes a lot of memory in unused regions, and b) introduces larger "stop-world"-pauses

2017-02-12 16:45:56 GMT <AFaust> yreg: That is only because when you pass 32G heap size the compressed oops feature is disabled and memory addresses are the real 64 bit addresses they need to be

2017-02-12 16:46:32 GMT <yreg> exactly

2017-02-12 16:47:25 GMT <AFaust> So yes, there is overhead that will be added when going to and above 32 G, but the effect is dependent on the structure / size of objects that are instantiated...

2017-02-12 16:48:07 GMT <AFaust> The example code linked to in the blog article is very aggressive in terms of forcing expensive object pointers

2017-02-12 16:49:10 GMT <AFaust> Anyway

2017-02-12 16:49:20 GMT <AFaust> Have you considered using the G1 GC algorithm?=

2017-02-12 16:50:20 GMT <AFaust> At least for Repository/Share it is much more efficient.... there is no more strict sizing for Eden, Survivor and Old Generation (as long as you don't force it), so issues with these should not occur anymore

2017-02-12 16:50:26 GMT <yreg> I am willing to experiment with it at least to see if it is any better

2017-02-12 16:50:55 GMT <yreg> the issue with eden space only occured on the solr side

2017-02-12 16:51:08 GMT <AFaust> For SOLR, some people claim that G1 GC can have side-effects in certain situations. Unfortuantely I have not yet had a chance to verify this with a production-grade system.

2017-02-12 16:51:25 GMT <yreg> and after tomcat crashed, it kept krashing whenever I restarted the service

2017-02-12 16:51:42 GMT <yreg> I ended up restarting the VM for it to work correctly again

2017-02-12 16:52:22 GMT <AFaust> Regarding SOLR + CMS - I usually have JVM settings that are adapted to increase relative size of eden/survivor in comparison to old generation, as most of memory in SOLR is usually needed for per-query data structures

2017-02-12 16:52:52 GMT <AFaust> -XX:NewRatio=1 -XX:SurvivorRatio=2 -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=80

2017-02-12 16:53:07 GMT <AFaust> ^^ my standard JVM settings for SOLR in addition to Xmx/Xms

2017-02-12 16:54:24 GMT <AFaust> Whhat are the settings you are using? Which of the various Alfresco documentations / blog post entries did you use as reference?

2017-02-12 16:54:25 GMT <yreg> Thanks for sharing this, will give it a run on this system and let you know how that goes !

2017-02-12 16:54:53 GMT <yreg> wait a sec, it was a wolleague who configured the servers

2017-02-12 16:56:27 GMT <yreg> -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ReservedCodeCacheSize=128m -Xmx=31744m -Xms=31744m

2017-02-12 16:59:42 GMT <yreg> I do not think I should be increasing Xmx and Xms to more than 32 G right now

2017-02-12 17:05:42 GMT <yreg> I do not have accurate measures sustaining this

2017-02-12 17:06:31 GMT <AFaust> I never understood the benefit of setting -XX:+UseParNewGC and -XX:ReservedCodeCacheSize

2017-02-12 17:06:53 GMT <AFaust> And I never encountered any issues not setting these (I have never used these)

2017-02-12 17:06:56 GMT <yregaieg> but I got the impression (from the graphs in our monitoring stack) that 31G worked better during reindex

2017-02-12 17:07:24 GMT <yregaieg> but this definitely deserve some deeper analysis

2017-02-12 17:07:33 GMT <AFaust> For SOLR I can understand how the OOP issue with 31G vs 32+G might be an issue

2017-02-12 17:07:58 GMT <AFaust> Because SOLR uses a lot of very small objects internally...

2017-02-12 17:08:29 GMT <AFaust> I still have to look up and understand how primitives / arrays of primitives work with OOP

2017-02-12 17:09:19 GMT *** yregaieg is now known as yreg

2017-02-12 17:12:33 GMT <AFaust> hehe UseParNewGC deprecated in JDK 8 and removed in JDK 9

2017-02-12 17:13:16 GMT <AFaust> damn - the guy who wrote that did not express himself very clearly

2017-02-12 17:13:23 GMT <yreg> Makes sense, by analysing graphs for heap size over time

2017-02-12 17:13:33 GMT <yreg> I get the impression that it is not effective

2017-02-12 17:13:46 GMT <AFaust> UseParNewGC is not deprecated in general - the guy attributed the remarks to the implicit use of CMS which is deprecated in 8 and removed in 9

2017-02-12 17:15:19 GMT <yreg> For alfresco side, what would be a typical config ?

2017-02-12 17:15:21 GMT <yreg> -XX:+UseG1GC -XX:MaxGCPauseMillis=250

2017-02-12 17:15:22 GMT <yreg> ?

2017-02-12 17:16:32 GMT <AFaust> Something like that, yes. I typically don't even bother with the max pause setting

2017-02-12 17:16:43 GMT <AFaust> The default value is quite reasonable

2017-02-12 17:17:40 GMT <AFaust> And since caches can be configured to proper sizes based on load tests / metrics to avoid exchausting the heap there should rarely be a case when longer GC pauses happen

2017-02-12 17:18:15 GMT <AFaust> Too bad it is the Lucene folk who warn against G1GC, even though SOLR folk don't seem to have ever encountered an issue with it

2017-02-12 17:18:15 GMT <AFaust> https://wiki.apache.org/solr/ShawnHeisey#G1_.28Garbage_First.29_Collector

2017-02-12 17:18:16 GMT <alfbot> Title: ShawnHeisey - Solr Wiki (at wiki.apache.org)

2017-02-12 17:19:12 GMT <AFaust> Lately I have started to set G1HeapRegionSize

2017-02-12 17:19:37 GMT <AFaust> Especially if I use a Xms that is not equal to Xmx

2017-02-12 17:20:08 GMT <AFaust> Because the default value is calculated based on Xms, which might not be ideal when the heap has grown to Xmx

2017-02-12 17:21:08 GMT <yreg> interesting

2017-02-12 17:22:20 GMT <yreg> Thank you for these valuable thoughts !

2017-02-12 17:22:21 GMT <AFaust> Some of the implications of G1GC and G1HeapRegionSize are also interesting for the question "when to shard"...

2017-02-12 17:23:26 GMT <yreg> hmm, that is not applicable in this case, but definitely interesting to know

2017-02-12 19:17:13 GMT <rootinshell> Hello eveyone

2017-02-12 19:17:21 GMT <rootinshell> got a question :)

End of Daily Log

The other logs are at http://esplins.org/hash_alfresco