BSS is hanging out or responding with delays

mbylica · Post by **mbylica** » Tue Sep 03, 2013 9:32 am

1. Brekeke Product Name and version:
BSS, 2.4.8.6, 3.2.3.6

2. Java version:
1.6.0_25, 1.6.0_45

3. OS type and the version:
linux centos 5.5

4. UA (phone), gateway or other hardware/software involved:
doesn't matter

5. Your problem:
We are having 1k concurrent calls (just sip proxy) running BSS 3.2.3.6 and in peak hours notifying problems with its basic functionality.
It looks like BSS is hanging out for some period of time (secs to minutes), not responding to any requests or with few seconds of delay. After a while BSS is coming back to life.
I do not have catalina logs storred as for now.
But sv logs shows sth like...

Just after reboot.

==============================================
svlistener: Exception: 09/03/13 16:16:01.463
java.lang.NullPointerException
com.brekeke.net.sip.common.ResendStat.getID (null:-1)
com.brekeke.net.sip.common.SIPpacketEx.clearResendStat (null:-1)
com.brekeke.net.sip.common.SIPthread.put (null:-1)
com.brekeke.net.sip.sv.SessionStat.put (null:-1)
com.brekeke.net.sip.sv.SIPsvListener.dispatch (null:-1)
com.brekeke.net.sip.sv.SIPsvListener.run (null:-1)
startup-time: Tue Sep 03 16:14:51.693 CEST 2013
life-length: 00:01:09.770
sip-packet-total: 768
stacked-packet: 1
stacked-packet-peak: 9
session-active-invite: 83
session-active-subscribe: 0
session-active-all: 83
session-total: 108
session-peak: 83
thread-tbl-size: 10
session-tbl-size: 301
session-per-sec: 12
registered-record: 0
route-tbl-size: 0
user-dir-size: 0
thread-active-cnt: 17
SIPex-thread-share: 10 / 100
SIP-error: 15
Internal-error: 1
java-use-memory: 61.68 MB (63157.06 KB 64672832 Byte)
java-free-memory: 418.95 MB (429002.94 KB 439299008 Byte)
java-total-memory: 480.62 MB (492160.00 KB 503971840 Byte)
java-max-memory: 480.62 MB (492160.00 KB 503971840 Byte)
svlistener: send response=500: warning: exception: java.lang.NullPointerException
at 09/03/13 16:16:01.464

later on...

java-use-memory: 150.51 MB (154120.60 KB 157819496 Byte)
java-free-memory: 250.93 MB (256951.40 KB 263118232 Byte)
java-total-memory: 401.44 MB (411072.00 KB 420937728 Byte)
java-max-memory: 455.12 MB (466048.00 KB 477233152 Byte)

java-use-memory: 313.07 MB (320584.44 KB 328278464 Byte)
java-free-memory: 48.99 MB (50167.56 KB 51371584 Byte)
java-total-memory: 362.06 MB (370752.00 KB 379650048 Byte)
java-max-memory: 455.12 MB (466048.00 KB 477233152 Byte)

java-use-memory: 170.27 MB (174357.04 KB 178541608 Byte)
java-free-memory: 4.98 MB (5098.96 KB 5221336 Byte)
java-total-memory: 175.25 MB (179456.00 KB 183762944 Byte)
java-max-memory: 455.12 MB (466048.00 KB 477233152 Byte)

I am running BSS with following arguments
-Xmx512m -Dfile.encoding=ISO-8859-2
BSS is exchanging AUTH/ACC radius packets with external billing system.

Do you guys had similiar problems?
Do you think that the problem i am facing is because of lack of memory?

I am about to try set -Xms8192m -Xmx24576m settings and reboot the BSS.

Unfortunately BSS 2.4.8.6 is running java/tomcat 32bit so max is 2048 here.

Thanks in advanced.

mbylica · Post by **mbylica** » Fri Sep 06, 2013 5:54 am

It looks like it was memory issue.

Tata · Post by **Tata** » Tue Sep 10, 2013 2:13 am

Hi mbylica,

Can you paste a SIP packet here?

According to the Exception message, I suppose there is a syntax issue in SIP packets.

For example, is there a "branch=" parameter in INVITE and its CANCEL?

Tata · Post by **Tata** » Tue Sep 10, 2013 2:29 am

For memory activities, you don't have to worry about it until the "java-use-memory" reaches the "java-max-memory".

The memory issue will not cause the above NullPointerException.

FYI:
Tuning of the [Performance Optimization (Proxy)] will reduce memory usages.

If you know how to execute a management command, execute "gc" command which performs a garbage-collection.

mbylica · Post by **mbylica** » Mon Sep 23, 2013 3:44 pm

Hi Tata,

First of all sorry for delay.
I am unable to copy&paste the INVITE or CANCEL right now, but i am sure that most of the calls were originated from well standarized MGW.
Maybe you are right and there are some requests that are not fully understandable from other sources - i will take a look on this.
But refering to what was said, it seems BSS is responding to such not fully compliant messages.

Regarding memory, that was exactly the effect i was getting and extending java memory solved the problem - now BSS is stable.
Inistal threads are set to 10 max is 20.

I am not guru in Java, so i will omit gc as for now

Again, thanks for feedback Tata.

Tata · Post by **Tata** » Tue Sep 24, 2013 4:40 pm

BSS will not accept CANCEL if it doesn't have the correct "branch" parameter but it will send an error response .

For tuning the memory:
Setting a larger value in the [Maximum Sessions per thread] will also save the memory usage.
It is the number of SIP sessions shared under each thread.

mbylica · Post by **mbylica** » Wed Sep 25, 2013 3:34 pm

Maximum set to 100

Thanks for clearing this up.