# Troubleshooting

# Automatic system repair

Use kctl rescue to automatically detect and fix most common issues on the server.

  1. Open terminal.
  2. Run:
kctl rescue
Does it show error "kctl: command not found"?

That means, you must to upgrade the server configuration first.


Server works slow, 502 errors

Open Maintenance — Status in Keitaro. Check the value in the line CPU Stolen.

If value greater than 0, that means the cloud server doesn't receive enough CPU time. You must report about that issue to the support of the cloud provider.

No enough space on the server

If server has 10-15% space, try Optimize the database, on other case, increase server size or delete all the statistics.

I increased RAM, but Keitaro don't see it.

You must upgrade server configuration.

How to understand that everything works well?

Keitaro shows yellow exclamation mark on the top if something is wrong. You may also check System Log.

Errors "Some internal error" or "See system log"

Open Maintenance → Logs → System log.

"RedisException. OOM command not allowed when used memory > maxmemory" error
  1. Go to Maintenance → Settings → System
  • Set a lower LP Token expire time.
  1. Set lower uniqueness expire time in the campaigns.

  2. Upgrade the server configuration.

Keitaro cannot refresh the license
  • If you changed /etc/resolve.conf, revert the changes.
  • If you changed firewall settings, check if it allows outgoing connections.
Got error 28 from storage engine

It means not enough disk space.

To show disc space in a server console, use the following command:

df -h
Redis is LOADING the dataset.

That solution deletes all the unprocessed statistics, but fixes the issue fast.

Run in server terminal:

rm -rf /var/lib/redis/*
systemctl restart redis
Database is corrupted

Mostly this occurs when MySQL database of your tracker crashed because of major failure in SSD read or write error (bad blocks, IO error, bad or defective SSD drive).

  1. In the terminal, run:

sudo touch /forcefsck
  1. Reboot the server:

  1. Start MySQL DB service in normal mode

systemctl start mysql
How to free inodes usage?

If your tracker started to work incorrectly, and you see messages about low disk space (while df -h showing you that you have enough disk space). In such case, possible that inodes usage on your server is close to its capacity. Inodes—are file descriptors in Linux, which store the attributes and disk block locations of the object's data. File-system object attributes may include metadata (times of last change, access, modification), as well as owner and permission data. When the number of inodes is close to 0 - that means your OS is insufficient of file descriptors ( inodes)

In such a case, you need to delete those files. For searching inodes, you need to use the following command:

df -i

We will receive the output of the structure of your file system with a list of inodes number directly. Now you need to follow the directory with the biggest number of inodes in your system—you need to go in a directory with the biggest inodes number and look for the folder with the biggest inodes number. Finally, you will find the folder with the biggest inode count number. After that—you will need to delete this folder with the following command:

rm -rf deletedfolder/ 

Where deletedfolder/ is the name of the folder with the biggest inode number which you need to delete. WARNING Please verify the folder which you are going to delete—your target is to delete the folder with most of the files descriptors (or biggest inode number) BUT not system files of your OS or a server. If you are not sure what are you doing. It is better to contact with technical support which will help you gladly to free inodes usage on your server. In other cases, you are risking to make your server unworkable.

Server works slowly

This could happen because of the lack of CPU resources for your virtual server. Run the command:

vmstat 1 5

As a result you'll see the following table:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
111  0      0 3438984  20280 603200    0    0    11    20   38   85 10  2 32  0 56
139  0      0 3435676  20280 603220    0    0     0     0  936  318 15  3  0  0 82
126  0      0 3434312  20280 603212    0    0     0     0  903  198 14  4  0  0 82
92   0      0 3450964  20280 603868    0    0     0     0  885  198 13  3  0  0 83
145  1      0 3452128  20280 604412    0    0   448     0  993  254 14  2  0  0 84

Pay attention to the last column st. If the value is more than 0, your VPS has insufficient resources. Contact the server provider in this case.

How do I check if my server has HDD or SSD?

Check the disks' names here:

cat /proc/scsi/scsi
Error "malformed worker response"
  1. Open SSH terminal.
  2. Run the following command:
cd /var/www/keitaro/application
find . -type f -name "*.php" -exec sed -i 's/?>//g' {} +

# Optimizing database

Stop Cron:

systemctl stop crond

Start Cron again:

systemctl start crond

# Deleting all the statistics

kctl run cli-php system:delete_all_stats --truncate-all-stats

After typing it in you will get the following warning:

[WARNING] Important! The following will happen:
           - old versions tables removal,
           - clicks stats to the exact date removal (if specified in parameters),
           - optimization of the clicks in the stats,
           - cleaning all the current stats completely (if specified in parameters).

           These inquiries will block the database
           during the implementation and require additional free space on a disk.

Type in yes and press ENTER to confirm.

Automatic cleaning

To turn on automatic data cleaning, set Stats store term in the settings.