# Production Deployment Guide

This guide provides comprehensive instructions for deploying the Multi-Tenant SaaS Platform to production environments.

## Prerequisites

### Infrastructure Requirements
- **Server**: Cloud VPS or dedicated server with minimum specifications
- **OS**: Ubuntu 20.04 LTS or CentOS 8+
- **RAM**: 16GB+ recommended
- **CPU**: 8+ cores recommended
- **Storage**: 200GB+ SSD storage
- **Network**: 1Gbps+ bandwidth

### Software Requirements
- **Python**: 3.9+
- **Node.js**: 16+
- **PostgreSQL**: 13+
- **Redis**: 6+
- **Nginx**: 1.18+
- **Supervisor**: Process management
- **Certbot**: SSL certificates
- **Fail2ban**: Security

### Malaysian Requirements
- **Domain**: Malaysian domain (.com.my, .my)
- **SSL**: Valid SSL certificate
- **Data Center**: Malaysian cloud region
- **Payment Gateway**: Malaysian payment providers
- **Compliance**: PDPA and KKM compliance

## Server Preparation

### 1. System Update and Security
```bash
# Update system packages
sudo apt update && sudo apt upgrade -y

# Install security packages
sudo apt install -y ufw fail2ban unattended-upgrades

# Configure firewall
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https
sudo ufw enable

# Configure automatic security updates
sudo dpkg-reconfigure -plow unattended-upgrades
```

### 2. Create Application User
```bash
# Create application user
sudo adduser --system --group --home /opt/multi-tenant-saas multi-tenant

# Set up sudo access for deployment
sudo visudo
# Add: multi-tenant ALL=(ALL) NOPASSWD:/usr/bin/systemctl restart gunicorn, /usr/bin/systemctl restart nginx
```

### 3. Install Required Software
```bash
# Install Python and development tools
sudo apt install -y python3-pip python3-venv python3-dev build-essential

# Install Node.js
curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash -
sudo apt install -y nodejs

# Install PostgreSQL
sudo apt install -y postgresql postgresql-contrib

# Install Redis
sudo apt install -y redis-server

# Install Nginx
sudo apt install -y nginx

# Install Supervisor
sudo apt install -y supervisor

# Install Certbot
sudo apt install -y certbot python3-certbot-nginx
```

## Database Setup

### 1. PostgreSQL Configuration
```bash
# Configure PostgreSQL
sudo -u postgres psql

# Create database and user
CREATE DATABASE multi_tenant_saas_prod;
CREATE USER multi_tenant_prod_user WITH PASSWORD 'strong-password-here';
GRANT ALL PRIVILEGES ON DATABASE multi_tenant_saas_prod TO multi_tenant_prod_user;
ALTER USER multi_tenant_prod_user CREATEDB;

# Configure PostgreSQL settings
sudo vim /etc/postgresql/13/main/postgresql.conf
# Update:
# shared_buffers = 256MB
# effective_cache_size = 1GB
# maintenance_work_mem = 64MB
# checkpoint_completion_target = 0.9
# wal_buffers = 16MB
# default_statistics_target = 100
# random_page_cost = 1.1
# effective_io_concurrency = 200
# work_mem = 4MB
# min_wal_size = 1GB
# max_wal_size = 4GB

# Restart PostgreSQL
sudo systemctl restart postgresql
```

### 2. Redis Configuration
```bash
# Configure Redis
sudo vim /etc/redis/redis.conf
# Update:
# maxmemory 512mb
# maxmemory-policy allkeys-lru
# save 900 1
# save 300 10
# save 60 10000

# Restart Redis
sudo systemctl restart redis
```

## Application Deployment

### 1. Deploy Application Code
```bash
# Switch to application user
sudo su - multi-tenant

# Clone repository
git clone https://github.com/your-org/multi-tenant-saas.git /opt/multi-tenant-saas
cd /opt/multi-tenant-saas

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt
pip install gunicorn psycopg2-binary

# Install frontend dependencies
cd frontend
npm install
npm run build
cd ..

# Set up environment
cp .env.production .env
vim .env
```

### 2. Environment Configuration
```bash
# Production environment variables
DEBUG=False
SECRET_KEY=your-production-secret-key-here
DATABASE_URL=postgresql://multi_tenant_prod_user:strong-password@localhost:5432/multi_tenant_saas_prod
REDIS_URL=redis://localhost:6379/0
ALLOWED_HOSTS=your-domain.com,www.your-domain.com
CORS_ALLOWED_ORIGINS=https://your-domain.com,https://www.your-domain.com

# Malaysian configuration
TIMEZONE=Asia/Kuala_Lumpur
CURRENCY=MYR
SST_RATE=0.06
DEFAULT_COUNTRY=Malaysia

# Security settings
SECURE_BROWSER_XSS_FILTER=True
SECURE_CONTENT_TYPE_NOSNIFF=True
SECURE_HSTS_INCLUDE_SUBDOMAINS=True
SECURE_HSTS_PRELOAD=True
SECURE_HSTS_SECONDS=31536000
SECURE_SSL_REDIRECT=True
SESSION_COOKIE_SECURE=True
CSRF_COOKIE_SECURE=True
```

### 3. Database Migrations
```bash
# Run migrations
python manage.py migrate --settings=config.production

# Create superuser
python manage.py createsuperuser --settings=config.production

# Load initial data
python manage.py load_initial_data --settings=config.production

# Collect static files
python manage.py collectstatic --settings=config.production --noinput
```

## Web Server Configuration

### 1. Nginx Configuration
```bash
# Create Nginx configuration
sudo vim /etc/nginx/sites-available/multi-tenant-saas
```

```nginx
server {
    listen 80;
    server_name your-domain.com www.your-domain.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name your-domain.com www.your-domain.com;

    ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;

    # Security headers
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    # Static files
    location /static/ {
        alias /opt/multi-tenant-saas/staticfiles/;
        expires 1y;
        add_header Cache-Control "public, immutable";
    }

    # Media files
    location /media/ {
        alias /opt/multi-tenant-saas/media/;
        expires 1y;
        add_header Cache-Control "public";
    }

    # Frontend
    location / {
        root /opt/multi-tenant-saas/frontend/build;
        try_files $uri $uri/ /index.html;
        expires 1h;
        add_header Cache-Control "public";
    }

    # API
    location /api/ {
        proxy_pass http://unix:/run/gunicorn.sock;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 30s;
        proxy_send_timeout 30s;
        proxy_read_timeout 30s;
    }

    # WebSocket
    location /ws/ {
        proxy_pass http://unix:/run/gunicorn.sock;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Health check
    location /health/ {
        proxy_pass http://unix:/run/gunicorn.sock;
        access_log off;
    }
}
```

### 2. Enable Site and Test
```bash
# Enable site
sudo ln -s /etc/nginx/sites-available/multi-tenant-saas /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
```

## Process Management

### 1. Gunicorn Configuration
```bash
# Create Gunicorn configuration
sudo vim /opt/multi-tenant-saas/gunicorn.conf.py
```

```python
bind = 'unix:/run/gunicorn.sock'
workers = 4
worker_class = 'sync'
worker_connections = 1000
timeout = 30
keepalive = 5
max_requests = 1000
max_requests_jitter = 100
preload_app = True
reload = False
daemon = False
user = 'multi-tenant'
group = 'multi-tenant'
raw_env = [
    'DJANGO_SETTINGS_MODULE=config.production',
    'PYTHONPATH=/opt/multi-tenant-saas',
]
```

### 2. Create Systemd Service
```bash
# Create Gunicorn service
sudo vim /etc/systemd/system/gunicorn.service
```

```ini
[Unit]
Description=Multi-Tenant SaaS Platform Gunicorn
After=network.target postgresql.service redis.service

[Service]
Type=notify
User=multi-tenant
Group=multi-tenant
WorkingDirectory=/opt/multi-tenant-saas
Environment=PATH=/opt/multi-tenant-saas/venv/bin
ExecStart=/opt/multi-tenant-saas/venv/bin/gunicorn --config gunicorn.conf.py config.wsgi:application
ExecReload=/bin/kill -s HUP $MAINPID
KillMode=mixed
TimeoutStopSec=5
PrivateTmp=true
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
```

### 3. Enable and Start Services
```bash
# Enable and start Gunicorn
sudo systemctl daemon-reload
sudo systemctl enable gunicorn
sudo systemctl start gunicorn

# Check status
sudo systemctl status gunicorn
```

## SSL Certificate Setup

### 1. Obtain SSL Certificate
```bash
# Stop Nginx temporarily
sudo systemctl stop nginx

# Obtain SSL certificate
sudo certbot certonly --standalone -d your-domain.com -d www.your-domain.com

# Start Nginx
sudo systemctl start nginx
```

### 2. Set Up Auto-Renewal
```bash
# Test auto-renewal
sudo certbot renew --dry-run

# Set up cron job for auto-renewal
sudo crontab -e
# Add: 0 12 * * * /usr/bin/certbot renew --quiet
```

## Malaysian Configuration

### 1. Payment Gateway Setup
```bash
# Configure Malaysian payment gateways
sudo vim /opt/multi-tenant-saas/config/payments.py
```

```python
PAYMENT_GATEWAYS = {
    'touch_n_go': {
        'enabled': True,
        'environment': 'production',
        'api_key': os.environ.get('TOUCH_N_GO_API_KEY'),
        'secret': os.environ.get('TOUCH_N_GO_SECRET'),
        'merchant_id': os.environ.get('TOUCH_N_GO_MERCHANT_ID'),
    },
    'grabpay': {
        'enabled': True,
        'environment': 'production',
        'api_key': os.environ.get('GRABPAY_API_KEY'),
        'secret': os.environ.get('GRABPAY_SECRET'),
        'merchant_id': os.environ.get('GRABPAY_MERCHANT_ID'),
    },
    'online_banking': {
        'enabled': True,
        'banks': ['maybank2u', 'cimb_clicks', 'rhbb'],
    }
}
```

### 2. SST Configuration
```bash
# Configure SST settings
sudo vim /opt/multi-tenant-saas/config/sst.py
```

```python
SST_SETTINGS = {
    'enabled': True,
    'rate': 0.06,
    'registration_number': os.environ.get('SST_REGISTRATION_NUMBER'),
    'currency': 'MYR',
    'invoice_prefix': 'SST',
    'tax_inclusive': True,
}
```

### 3. Timezone and Localization
```bash
# Set system timezone
sudo timedatectl set-timezone Asia/Kuala_Lumpur

# Configure application timezone
sudo vim /opt/multi-tenant-saas/config/settings.py
```

```python
TIME_ZONE = 'Asia/Kuala_Lumpur'
LANGUAGE_CODE = 'en-my'
USE_I18N = True
USE_L10N = True
USE_TZ = True
```

## Monitoring and Logging

### 1. Application Logging
```bash
# Create log directory
sudo mkdir -p /var/log/multi-tenant-saas
sudo chown multi-tenant:multi-tenant /var/log/multi-tenant-saas

# Configure logging
sudo vim /opt/multi-tenant-saas/config/logging.py
```

```python
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
            'style': '{',
        },
        'simple': {
            'format': '{levelname} {message}',
            'style': '{',
        },
    },
    'handlers': {
        'file': {
            'level': 'INFO',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/var/log/multi-tenant-saas/django.log',
            'maxBytes': 10485760,  # 10MB
            'backupCount': 5,
            'formatter': 'verbose',
        },
        'console': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
            'formatter': 'simple',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file', 'console'],
            'level': 'INFO',
            'propagate': False,
        },
        'multi_tenant_saas': {
            'handlers': ['file', 'console'],
            'level': 'INFO',
            'propagate': False,
        },
    },
}
```

### 2. System Monitoring
```bash
# Install monitoring tools
sudo apt install -y htop iotop nethogs

# Set up log rotation
sudo vim /etc/logrotate.d/multi-tenant-saas
```

```
/var/log/multi-tenant-saas/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 multi-tenant multi-tenant
    postrotate
        systemctl reload gunicorn
    endscript
}
```

## Backup and Recovery

### 1. Database Backup
```bash
# Create backup script
sudo vim /opt/multi-tenant-saas/scripts/backup-database.sh
```

```bash
#!/bin/bash
BACKUP_DIR="/opt/multi-tenant-saas/backups"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/database_backup_$DATE.sql"

# Create backup directory
mkdir -p $BACKUP_DIR

# Create database backup
pg_dump -h localhost -U multi_tenant_prod_user -d multi_tenant_saas_prod -f $BACKUP_FILE

# Compress backup
gzip $BACKUP_FILE

# Keep only last 30 days of backups
find $BACKUP_DIR -name "*.sql.gz" -mtime +30 -delete

echo "Database backup completed: $BACKUP_FILE.gz"
```

### 2. Automated Backups
```bash
# Make backup script executable
sudo chmod +x /opt/multi-tenant-saas/scripts/backup-database.sh

# Set up cron job for daily backups
sudo crontab -e
# Add: 0 2 * * * /opt/multi-tenant-saas/scripts/backup-database.sh
```

## Security Hardening

### 1. Application Security
```bash
# Set proper file permissions
sudo chown -R multi-tenant:multi-tenant /opt/multi-tenant-saas
sudo chmod -R 750 /opt/multi-tenant-saas
sudo chmod 600 /opt/multi-tenant-saas/.env

# Secure sensitive files
sudo chmod 600 /etc/nginx/sites-available/multi-tenant-saas
sudo chmod 600 /etc/systemd/system/gunicorn.service
```

### 2. Database Security
```bash
# Configure PostgreSQL security
sudo -u postgres psql

# Remove public access
REVOKE ALL PRIVILEGES ON DATABASE multi_tenant_saas_prod FROM PUBLIC;
REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM PUBLIC;
REVOKE ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public FROM PUBLIC;
REVOKE ALL PRIVILEGES ON SCHEMA public FROM PUBLIC;

# Grant only to application user
GRANT CONNECT ON DATABASE multi_tenant_saas_prod TO multi_tenant_prod_user;
GRANT USAGE ON SCHEMA public TO multi_tenant_prod_user;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO multi_tenant_prod_user;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO multi_tenant_prod_user;
```

## Performance Optimization

### 1. Database Optimization
```bash
# Create database indexes
sudo -u postgres psql -d multi_tenant_saas_prod

# Create indexes for common queries
CREATE INDEX CONCURRENTLY idx_core_tenant_schema_name ON core_tenant(schema_name);
CREATE INDEX CONCURRENTLY idx_core_user_username ON core_user(username);
CREATE INDEX CONCURRENTLY idx_core_user_email ON core_user(email);
CREATE INDEX CONCURRENTLY idx_core_user_tenant ON core_user(tenant_id);

# Analyze tables for better query planning
ANALYZE;
```

### 2. Application Optimization
```bash
# Configure Django settings for production
sudo vim /opt/multi-tenant-saas/config/production.py
```

```python
# Production optimizations
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.redis.RedisCache',
        'LOCATION': 'redis://127.0.0.1:6379/1',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
        }
    }
}

# Session configuration
SESSION_ENGINE = "django.contrib.sessions.backends.cache"
SESSION_CACHE_ALIAS = "default"

# Email configuration
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'smtp.your-email-provider.com'
EMAIL_PORT = 587
EMAIL_USE_TLS = True
EMAIL_HOST_USER = os.environ.get('EMAIL_HOST_USER')
EMAIL_HOST_PASSWORD = os.environ.get('EMAIL_HOST_PASSWORD')
```

## Testing and Verification

### 1. Health Checks
```bash
# Application health check
curl -f https://your-domain.com/health/ || exit 1

# Database connectivity
sudo -u multi-tenant psql -h localhost -d multi_tenant_saas_prod -c "SELECT 1;" || exit 1

# Redis connectivity
redis-cli ping || exit 1
```

### 2. Load Testing
```bash
# Install load testing tools
pip install locust

# Create load test script
sudo vim /opt/multi-tenant-saas/load-test.py
```

```python
from locust import HttpUser, task, between

class WebsiteUser(HttpUser):
    wait_time = between(1, 5)

    def on_start(self):
        self.client.get("/api/v1/health/")

    @task(3)
    def view_home(self):
        self.client.get("/")

    @task(1)
    def api_call(self):
        self.client.get("/api/v1/core/tenants/")
```

### 3. Security Testing
```bash
# Run security checks
pip install bandit safety
bandit -r /opt/multi-tenant-saas/backend/src/
safety check

# SSL security test
openssl s_client -connect your-domain.com:443 -servername your-domain.com | openssl x509 -noout -dates
```

## Deployment Checklist

- [ ] System requirements met
- [ ] Software dependencies installed
- [ ] Database configured and secured
- [ ] Application code deployed
- [ ] Environment variables set
- [ ] SSL certificate obtained
- [ ] Nginx configured
- [ ] Gunicorn service running
- [ ] Payment gateways configured
- [ ] Malaysian compliance settings
- [ ] Security hardening completed
- [ ] Monitoring and logging set up
- [ ] Backup procedures implemented
- [ ] Performance optimizations applied
- [ ] Health checks passing
- [ ] Load testing completed
- [ ] Security testing completed

## Support and Maintenance

### Regular Maintenance
- Daily: Check logs and monitor performance
- Weekly: Review security updates and patches
- Monthly: Database maintenance and optimization
- Quarterly: Security audit and compliance review
- Annually: Full system review and upgrade planning

### Emergency Procedures
- Application failure: Check logs, restart services
- Database issues: Restore from backup, contact support
- Security incident: Follow incident response plan
- Performance issues: Scale resources, optimize queries

### Support Contacts
- Technical Support: support@yourplatform.com
- Emergency Support: emergency@yourplatform.com
- Security Issues: security@yourplatform.com
- Sales Inquiries: sales@yourplatform.com