# monitor-restart

**Repository Path**: xskywalker/monitor-restart

## Basic Information

- **Project Name**: monitor-restart
- **Description**:  Java Process Monitor & Auto Restart System
- **Primary Language**: Shell
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-07-08
- **Last Updated**: 2025-08-21

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Java Process Monitor & Auto Restart System

A robust monitoring system for Java processes that automatically restarts processes when memory or CPU usage exceeds configured thresholds. Perfect for maintaining service availability and preventing resource exhaustion.

## 🚀 Features

- ✅ **Multi-process monitoring** - Monitor multiple processes simultaneously
- ✅ **Flexible thresholds** - Configure memory and/or CPU usage limits
- ✅ **Smart restart protection** - Prevents frequent restarts with configurable intervals
- ✅ **Continuous CPU detection** - Requires 3 consecutive high CPU readings to prevent false positives
- ✅ **Real-time CPU monitoring** - Uses top command for accurate real-time CPU usage data
- ✅ **Tolerance window** - 5-minute window for CPU spike detection with automatic reset
- ✅ **Comprehensive logging** - Detailed monitoring and restart history
- ✅ **DingTalk notifications** - Real-time alerts via DingTalk bot (optional)
- ✅ **Easy deployment** - Automated installation script included
- ✅ **Cron integration** - Ready for scheduled monitoring

## 📁 Project Structure

```
monitor-restart/
├── process_monitor.sh      # Main monitoring script
├── monitor_config.conf     # Process monitoring configuration
├── install_monitor.sh      # Automated installation script
├── .env                    # (Optional) Environment variables for DingTalk
├── README.md              # This file
├── monitor_usage.md       # Detailed usage documentation (Chinese)
└── logs/                  # Log directory (auto-created)
    ├── process_monitor.log     # Main log file
    ├── restart_history.log     # Restart history
    ├── .last_restart_*         # Per-process restart timestamps
    └── .cpu_detection_*        # CPU detection state files
```

## ⚡ Quick Start

### Option 1: Automated Installation (Recommended)

```bash
# 1. Clone or download the project
cd /path/to/monitor-restart

# 2. Run the installation script
./install_monitor.sh
```

### Option 2: Manual Setup

```bash
# 1. Set executable permissions
chmod +x process_monitor.sh install_monitor.sh

# 2. Edit configuration file
vim monitor_config.conf

# 3. Test run
./process_monitor.sh -t

# 4. Setup cron job
crontab -e
# Add: */1 * * * * /path/to/process_monitor.sh > /dev/null 2>&1
```

## ⚙️ Configuration

### Process Monitoring Configuration (`monitor_config.conf`)

**Format:**
```
process_name:memory_threshold:cpu_threshold:startup_script:work_directory:min_restart_interval
```

**Examples:**
```bash
# Monitor both memory (85%) and CPU (90%)
my_service_a:85:90:/apps/service_a/startup.sh:/apps/service_a:5

# Monitor memory only (80%), CPU disabled
my_service_b:80:0:/apps/service_b/startup.sh:/apps/service_b:10

# Monitor CPU only (95%), memory disabled  
my_service_c:0:95:/apps/service_c/restart.sh:/apps/service_c:3

# Using empty values (same as 0)
my_service_d::95:/apps/service_d/run.sh:/apps/service_d:5
```

**Parameters:**

| Parameter | Description | Example |
|-----------|-------------|---------|
| process_name | Process name as shown in `ps` command | `my_service` |
| memory_threshold | Memory usage threshold % (0 to disable) | `85` |
| cpu_threshold | CPU usage threshold % (0 to disable) | `90` |
| startup_script | Full path to restart script | `/apps/service_a/startup.sh` |
| work_directory | Working directory for script execution | `/apps/service_a` |
| min_restart_interval | Minimum interval between restarts (minutes) | `5` |

### DingTalk Notifications (Optional)

Create a `.env` file in the project root to enable DingTalk notifications:

```env
dingtalk_webhook_url="https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN"
dingtalk_secret="YOUR_SECRET_KEY"
```

> **Note:** If the `.env` file doesn't exist or lacks these variables, the script will silently skip notifications without affecting normal monitoring.

## 🛠️ Usage

### Basic Commands

```bash
# Show help
./process_monitor.sh -h

# Run with default configuration
./process_monitor.sh

# Specify custom config file
./process_monitor.sh -c /path/to/config.conf

# Specify custom log directory
./process_monitor.sh -l /path/to/logs

# Test mode (check only, no restart)
./process_monitor.sh -t
```

### Cron Setup

```bash
# Edit crontab
crontab -e

# Check every minute
*/1 * * * * /path/to/process_monitor.sh > /dev/null 2>&1

# Check every 2 minutes
*/2 * * * * /path/to/process_monitor.sh > /dev/null 2>&1

# Check every 5 minutes
*/5 * * * * /path/to/process_monitor.sh > /dev/null 2>&1
```

## 📊 Monitoring & Logs

### Log Files

- **Main Log** (`logs/process_monitor.log`) - All monitoring activities
- **Restart History** (`logs/restart_history.log`) - All restart events
- **Restart Timestamps** (`logs/.last_restart_*`) - Per-process restart times
- **CPU Detection State** (`logs/.cpu_detection_*`) - CPU detection history and state files

### View Logs

```bash
# Real-time monitoring
tail -f logs/process_monitor.log

# View restart history
cat logs/restart_history.log

# Search specific process logs
grep "my_service" logs/process_monitor.log
```

### Sample Log Output

```
[2024-01-15 14:30:01] [INFO] ========== Starting Process Monitor ==========
[2024-01-15 14:30:01] [INFO] Monitoring process: my_service_a (Memory: 85%, CPU: 90%)
[2024-01-15 14:30:01] [INFO] Process my_service_a current memory usage: 9%
[2024-01-15 14:30:01] [INFO] Process my_service_a current CPU usage: 12%
[2024-01-15 14:30:01] [INFO] Process my_service_a all metrics normal
[2024-01-15 14:30:01] [INFO] ========== Process Monitor Complete ==========

[2024-01-15 14:31:01] [INFO] Process my_service_a current CPU usage: 95%
[2024-01-15 14:31:01] [INFO] Process my_service_a CPU detection status: 1 consecutive times over threshold, current 95%, threshold 90%
[2024-01-15 14:31:01] [INFO] Process my_service_a CPU瞬时超过阈值，但未达到重启条件

[2024-01-15 14:32:01] [INFO] Process my_service_a current CPU usage: 130%
[2024-01-15 14:32:01] [INFO] Process my_service_a CPU detection status: 3 consecutive times over threshold, current 130%, threshold 90%
[2024-01-15 14:32:01] [WARN] Process my_service_a 连续3次CPU超过阈值，准备重启
[2024-01-15 14:32:01] [WARN] Process my_service_a 因 CPU使用率 (130%) 连续超过阈值 (90%)，准备重启
```

## 🔧 Troubleshooting

### Common Issues

1. **Process Not Found**
   ```
   [ERROR] Unable to get memory usage for process my_service
   [WARN] Unable to get CPU usage for process my_service, skipping CPU check.
   ```
   - **Check process name**: Verify the process name in `monitor_config.conf`
   - **Confirm process running**: Use `ps aux | grep 'process_name'`

2. **Startup Script Failed**
   ```
   [ERROR] Startup script does not exist: /path/to/startup.sh
   ```
   - Check if the startup script path is correct
   - Ensure the script has execute permissions

3. **DingTalk Notification Failed**
   ```
   [ERROR] DingTalk notification failed. Response code: 400
   ```
   - **Check `.env` file**: Verify `dingtalk_webhook_url` and `dingtalk_secret`
   - **Check DingTalk settings**: Ensure "Sign" security setting is enabled
   - **Check network**: Confirm server can reach `oapi.dingtalk.com`

4. **CPU Detection Not Triggering Restart**
   ```
   [INFO] Process my_service CPU瞬时超过阈值，但未达到重启条件
   ```
   - **Expected behavior**: The system requires 3 consecutive high CPU readings
   - **Check detection history**: View `logs/.cpu_detection_*` files
   - **Monitor logs**: Look for consecutive detection messages
   - **Verify thresholds**: Ensure CPU threshold is appropriate for your workload

5. **Frequent CPU Threshold Alerts**
   ```
   [INFO] Process my_service CPU detection status: 1 consecutive times over threshold
   ```
   - **Temporary spikes**: Normal behavior, system prevents false restarts
   - **Adjust thresholds**: Consider raising CPU threshold if too sensitive
   - **Monitor patterns**: Check if spikes correlate with specific activities (e.g., cron jobs)

### Debug Commands

```bash
# Test mode
./process_monitor.sh -t

# Check process manually
ps aux | grep my_service

# Check real-time CPU usage (same as monitoring script)
top -b -n 1 -p $(pgrep my_service) | tail -n +8

# Check CPU detection state
cat logs/.cpu_detection_my_service

# Verify startup script
cd /apps/service_a
./startup.sh

# Check permissions
ls -la process_monitor.sh
ls -la /apps/service_a/startup.sh

# Monitor CPU usage in real-time
watch -n 1 "top -b -n 1 -p \$(pgrep my_service) | tail -n +8 | awk '{print \$9}'"

# Clean up old detection state files
find logs/ -name ".cpu_detection_*" -mtime +1 -delete
```

## 🎛️ Advanced Features

### Smart CPU Detection System

The system implements an intelligent CPU detection mechanism to prevent false positives from temporary spikes:

**Key Features:**
- **Continuous Detection**: Requires 3 consecutive high CPU readings before triggering a restart
- **Real-time Monitoring**: Uses `top` command for accurate real-time CPU usage (not averaged values)
- **Tolerance Window**: 5-minute window for spike detection with automatic reset
- **State Management**: Tracks detection history in `logs/.cpu_detection_*` files
- **Auto-recovery**: Resets detection state when CPU returns to normal levels

**Detection Logic:**
1. **First Detection**: CPU > threshold → Record detection (count=1)
2. **Second Detection**: CPU > threshold → Increment count (count=2)
3. **Third Detection**: CPU > threshold → Trigger restart (count=3)
4. **Normal CPU**: CPU ≤ threshold → Reset detection state
5. **Timeout**: >5 minutes between detections → Reset count to 1

**Benefits:**
- Prevents restarts from brief CPU spikes (e.g., garbage collection)
- Maintains detection for genuinely persistent high CPU issues
- Reduces unnecessary service interruptions
- Provides detailed detection logging for troubleshooting

### Built-in DingTalk Notifications

When both conditions are met, the script automatically sends Markdown-formatted notifications to your specified DingTalk group before restarting a process:

1. `.env` file exists in the project root
2. `.env` file contains valid `dingtalk_webhook_url` and `dingtalk_secret`

Notifications include hostname, process name, **specific trigger reason** (memory limit, CPU limit, or both), current usage, thresholds, and restart time.

### Custom Notification Methods

To add other notification methods (e.g., email), modify the `restart_process` function in `process_monitor.sh`:

```bash
restart_process() {
    # ... existing code ...
    
    # Add your custom notification logic here
    # Example: Send email notification
    echo "Process $process_name restarted due to high resource usage" | \
        mail -s "Process Restart Alert" admin@example.com
    
    # ... existing code ...
}
```

## ⚠️ Important Notes

1. **Permissions** - Script needs permission to kill processes (usually root or process owner)
2. **System Load** - Avoid setting thresholds too low or monitoring too frequently
3. **Data Safety** - Restarts interrupt ongoing operations; consider graceful shutdown mechanisms
4. **Log Management** - Regularly clean up log files and state files to prevent disk space issues
5. **Testing** - Thoroughly test in non-production environment before deployment
6. **CPU Detection** - The 3-consecutive-detection mechanism prevents false positives but may delay response to genuine issues
7. **Multi-core Systems** - CPU usage is capped at 100% to avoid threshold comparison issues in multi-core environments

## 📋 Requirements

- **Operating System**: Linux/Unix with bash
- **Dependencies**: `ps`, `top`, `curl`, `openssl`, `base64` (usually pre-installed)
- **Permissions**: Ability to kill and restart target processes
- **Optional**: DingTalk bot with webhook URL and secret for notifications

## 🔄 Version Updates

When updating the monitoring script:

1. Backup current configuration and logs
2. Update script files
3. Re-run installation script if needed
4. Verify functionality

## 📝 License

This project is open source. Feel free to modify and distribute according to your needs.

## 🤝 Contributing

Contributions are welcome! Please feel free to submit issues, feature requests, or pull requests.

---

For detailed Chinese documentation, see [monitor_usage.md](monitor_usage.md).