Advanced Troubleshooting

Quick diagnostics

Be sure to look in the oasis_admin.log (/var/log/oasis) first for any signs of trouble. Look for the hourly reloads and make sure you don't see anything that suggests problems. If you're using OASIS Enterprise, rely on the master server's admin log first.

Also, look in the Apache logs for anything unusual, like 404 errors or the like.

Disk space

Disks filling up are a common problem with any sort of high-traffic Internet server. OASIS is not immune to this. You need to implement good log rotation and archiving to prevent your disks from filling up.

If OASIS is not behaving as expected, checking to see that there is adequate disk space is an easy first diagnostic test. Use du to see how much room is on each partition.

On a high traffic OASIS server, you might run into trouble with the following growing without bound:

oasis logs (/var/log/oasis)
apache logs (/var/log/httpd, usually)

If you are in need of immediate space, you can delete anything under /var/log/oasis/YYYY, which are archived log files.

Shared memory

Another common source of problems with OASIS is misconfiguration of shared memory. The most common problem is that the shared memory segments are owned by the wrong user.

Use ipcs to look at the shared memory segments, their size, and their owner. You want them all to be owned by the user who runs your Web server. If they're not owned properly, you can delete them (use ipcrm and then run hourly_maint.php start to recreate them.

Another common problem is that the shared memory segments are too small for the required data. You'll see log entries in the admin log to this effect. If this is the case, you should go into the Admin/Preferences page to change the shared memory segment sizes (be sure not to exceed your OS's maximum shared memory size (defined in /proc/sys/kernel/shmmax under RedHat Linux). You'll need to stop and restart OASIS (use hourly_maint.php stop and hourly_maint.php start (as the Apache user, of course).

PHP Configuration

PHP must be configured carefully to accomodate OASIS. Edit /etc/php.ini to change PHP configuration.

On an Enterprise server, the master server communicates with the slaves via HTTP POST. Make sure that post_max_size is big enough to accomodate the rather large files sent from master to slave. Look at the /tmp/OASIS* files to see how large they can get. Note that these files can get bigger as you add more sections and creatives to your server. Make sure you have lots of headroom here.

When the slave servers parse the XML sent from the master server, they parse it into an XML DOM, which uses a lot of memory. Be sure that your PHP processes are allowed to use enough memory. Edit the memory_limit parameter in php.ini.

Be sure to restart Apache after editing php.ini.

Apache Configuration

Much like the post_max_size PHP configuration variable, Apache can limit the size of accepted requests. Make sure that LimitRequestBody is not set too small in your Apache config files. Under RedHat, this is set in /etc/httpd/conf.d/php.conf.

MySQL Issues

Finally, MySQL can present problems. It is used on all OASIS servers, even slave servers in an Enterprise configuration. You must have MySQL functioning properly on all servers.

Check to make sure that MySQL is running. Use /sbin/service mysqld status to check.

Make sure that you can connect to MySQL and use the oasis database. Check /etc/httpd/conf/oasis_httpd.conf for the username and password being used to connect to MySQL. Use those in the following command:

mysql -u USERNAME -p

Enter the password. Now at the mysql prompt, type this:

use oasis

If you can't connect, you likely need to fix oasis_httpd.conf so that it contains the right username/password. If you don't know the right username and password, you'll need to connect to MySQL's mysql database to set the username and password.

Other Diagnostic Tools

OASIS ships with a number of diagnostic delivery scripts that let you see the internals of the delivery engine at work. Look in the templates directory when you expand the source tarball. The diagnostic scripts end with "-d.php". Copy those into the directory where your delivery scripts live, and then replace a normal delivery URL with the URL to one of these scripts (keep all the CGI parameters, of course). You'll see copious debugging output that will be helpful to determine why you don't see what you expect.

Common reasons for creatives not showing up as expected:

No creatives assigned to the specified section
No creatives of the right dimensions assigned to the specified section
Delivery controls that are too restrictive

Another helpful diagnostic tool is showshm.php, which is found in the utils directory when you expand the source tarball. Use it to see exactly what is in shared memory. You can, of course, see these things via the Web interface, but sometimes (especially on slave servers in an Enterprise configuration), it is helpful to see them from the command line.

Adding a new slave to an Enterprise cluster

When you add a new server, you must call master_reload.php with the start command-line argument. Once this is called, the shared memory segments are now defined properly on the new slave, and you should immediately call master_reload.php again without the start argument.