Debugging a mystery Unix server

I was given an Unix server to administer with no runbook. The immediate task was to find the application listening on port 8888, make some modifications, and restart.

Step 1: Find process listening on port 8888

The netstat command shows the network statistics on a Unix server, including which process is listening to what port

netstat -plan | grep 8888

The output is

tcp     0   0 :::8888   :::*    LISTEN      2700/java 

This shows that a java program with Process ID (PID) 2700 is listening on port 8888

Step 2: Find process info of PID 2700

The ps command shows the current process snapshot.

ps -fp 2700

The output is

oracle  2700   1  0 18:55 ?     0:00:02 
java -Dapex.port=8888 -jar /home/oracle/listener/apex.war

PID 2700 is a Java program executing apex.war as the Unix user oracle. A quick Google search reveals that Oracle publishes a product called Oracle Apex.

Step 3: Find the configuration and log files

The lsof command lists open files by PID. This can help us find configuration files and log directories.

lsof -p 2700

The output is

java    2700 oracle    2w   REG 3,1 1700 1796967 /tmp/apex_listener.log
java    2700 oracle    3r   REG. 3,65. 9828505  134585 /home/oracle/listener/apex.war
java    2700 oracle    4r   REG 3,1 51796975  361047 /usr/java/jdk1.6.0_20/jre/lib/rt.jar
java    2700 oracle    5r   REG. 3,65  186902  132940 /home/oracle/config/apex.xml

With a bit of deductive reasoning, we know the configuration file is at /home/oracle/config/apex.xml and the log file is located at /tmp/apex_listener.log

Step 4: Stop and start the server

After editing the apex.xml configuration file, most java programs require a restart to apply the new settings. If you scroll back up to Step 2 and look at the ps output, the number after PID 2700 in the third column is the parent PID 1.

The command

ps -fp 1

gives the output

root    1     0  0 18:54 ?    00:00:00 init [5]

The init command is the process that executes when Unix first boots up.  The startup scripts are in directory /etc/init.d

The find command to find which script stops and starts Oracle Apex

find /etc/init.d -type f -exec grep -Hi apex {} \;

The output is

/etc/init.d/oracle-apex:   su - oracle -c "/home/oracle/apex/listener.sh start"

/etc/init.d/oracle-apex:   su - oracle -c "/home/oracle/apex/listener.sh restart"

/etc/init.d/oracle-apex:   su - oracle -c "/home/oracle/apex/listener.sh stop"

Reading the oracle-apex Bash script, the command to restart Oracle Apex is

/etc/init.d/oracle-apex restart