Linux Service Operations

This page is a quick operational reference for Linux service management and server maintenance. We cover systemd and systemctl, service creation and removal, Nginx site cleanup, PID checks, journalctl, Docker logs, cron, and cron versus Celery.

Overview

• Use systemctl to control systemd-managed services.
• Use daemon-reload after creating, editing, or deleting systemd unit files.
• Use restart for app services after code or environment changes.
• Use nginx -t and systemctl reload nginx after Nginx config changes.
• Use journalctl for systemd service logs.
• Use docker compose logs for container application logs.
• Use cron for simple time-based shell commands.
• Use Celery for application background jobs, retries, queues, and scheduled app tasks.

Mental model

Linux kernel
  -> systemd
    -> services
      -> nginx
      -> docker
      -> ssh
      -> cron or cronie
      -> custom app services

systemctl
  -> command tool used to control systemd

journalctl
  -> command tool used to read logs collected by systemd-journald

• systemd is the service manager daemon.
• systemctl is the command-line control tool.
• journalctl is the command-line log reader.
• Nginx, Docker, cron, and custom apps can all run as services managed by systemd.

Check PID 1

PID 1 is usually systemd on modern Arch and Ubuntu systems. It may appear as /sbin/init because that path points to systemd.

ps -p 1 -o pid,comm,args
ls -l /sbin/init
readlink -f /sbin/init

Example output:

PID COMMAND  COMMAND
1   systemd  /sbin/init

/sbin/init -> ../lib/systemd/systemd
/usr/lib/systemd/systemd

Common systemctl commands

systemctl status nginx
systemctl status docker
systemctl status my-app.service

sudo systemctl start my-app.service
sudo systemctl stop my-app.service
sudo systemctl restart my-app.service

sudo systemctl enable my-app.service
sudo systemctl disable my-app.service

sudo systemctl daemon-reload
sudo systemctl reset-failed

Command	Use
status	Show service state, recent logs, and main process
start	Start a stopped service
stop	Stop a running service
restart	Stop and start the service process
reload	Ask a service to re-read its own config if supported
enable	Start service automatically on boot
disable	Prevent service from starting on boot
daemon-reload	Reload systemd unit definitions
reset-failed	Clear old failed-state records

daemon-reload, restart, and reload

• daemon-reload refreshes systemd unit definitions. It does not restart running services.
• restart restarts the actual service process.
• reload asks a running service to re-read its own config if the service supports reload.

sudo systemctl daemon-reload
sudo systemctl restart my-app.service
sudo systemctl reload nginx

Use both daemon-reload and restart after changing a .service file.

sudo nano /etc/systemd/system/my-app.service
sudo systemctl daemon-reload
sudo systemctl restart my-app.service
sudo systemctl status my-app.service

Use only restart when the app code changed but the .service file did not change.

sudo systemctl restart my-app.service
sudo systemctl status my-app.service

Create a systemd service

Create a service file under /etc/systemd/system for custom application services.

sudo nano /etc/systemd/system/my-app.service

Example service file:

[Unit]
        Description=My App Service
        After=network.target

        [Service]
        WorkingDirectory=/home/ubuntu/my-app
        ExecStart=/home/ubuntu/my-app/.venv/bin/python app.py
        Restart=always
        RestartSec=5
        User=ubuntu
        EnvironmentFile=/home/ubuntu/my-app/.env

        [Install]
        WantedBy=multi-user.target

Load the new definition, start it, enable it on boot, and inspect status.

sudo systemctl daemon-reload
sudo systemctl start my-app.service
sudo systemctl enable my-app.service
sudo systemctl status my-app.service

Update a systemd service

If the service file changes, reload systemd definitions and restart the service process.

sudo nano /etc/systemd/system/my-app.service
sudo systemctl daemon-reload
sudo systemctl restart my-app.service
sudo systemctl status my-app.service

If only application code changed, restart the service without daemon-reload.

sudo systemctl restart my-app.service
sudo systemctl status my-app.service

Remove a systemd service

Stop the service, disable boot startup, confirm the unit file path, remove the unit file, reload systemd definitions, and clear failed state if needed.

sudo systemctl stop service-name.service
sudo systemctl disable service-name.service
sudo systemctl cat service-name.service
sudo rm /etc/systemd/system/service-name.service
sudo systemctl daemon-reload
sudo systemctl reset-failed
systemctl status service-name.service

Expected final output:

Unit service-name.service could not be found.

Remove multiple custom services

Use one command per phase when removing a group of related services.

sudo systemctl stop app-backend.service app-worker.service app-beat.service app-frontend.service

sudo systemctl disable app-backend.service app-worker.service app-beat.service app-frontend.service

sudo rm /etc/systemd/system/app-backend.service
sudo rm /etc/systemd/system/app-worker.service
sudo rm /etc/systemd/system/app-beat.service
sudo rm /etc/systemd/system/app-frontend.service

sudo systemctl daemon-reload
sudo systemctl reset-failed

Confirm no matching units remain.

systemctl list-unit-files | grep app
systemctl list-units --all | grep app

Nginx as a systemd service

• Nginx is installed server software.
• nginx.service is the systemd service that runs Nginx.
• systemctl manages the Nginx service lifecycle.
• Nginx usually acts as the public entry point for web traffic.

sudo systemctl status nginx
sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx
sudo systemctl reload nginx
sudo systemctl enable nginx
sudo systemctl disable nginx

systemctl cat nginx

Nginx config locations

/etc/nginx/nginx.conf
/etc/nginx/sites-available/
/etc/nginx/sites-enabled/

• sites-available stores site config files.
• sites-enabled stores symlinks to active site configs.
• A file in sites-available is inactive unless it is linked from sites-enabled.

ls /etc/nginx/sites-available
ls -l /etc/nginx/sites-enabled

Remove an Nginx site config

Remove the enabled symlink first, then remove the available config file. Test before reloading Nginx.

ls -l /etc/nginx/sites-enabled | grep example-site

sudo rm -f /etc/nginx/sites-enabled/example-site.conf
sudo rm -f /etc/nginx/sites-available/example-site.conf

sudo nginx -t
sudo systemctl reload nginx

Expected nginx -t output:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Default Nginx site

• The default file can stay in sites-available as a reference.
• It is inactive if it is not linked from sites-enabled.
• Remove only the sites-enabled/default symlink if the default site is active and not needed.

ls -l /etc/nginx/sites-enabled

sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t
sudo systemctl reload nginx

Why Nginx is used in production

In development, apps are often accessed directly on localhost ports. In production, users usually access Nginx on ports 80 and 443, then Nginx forwards traffic to internal app ports.

Development:

Browser
  -> http://localhost:3000
  -> http://localhost:8000

Production:

Browser
  -> https://example.com
    -> Nginx on port 443
      -> frontend app on localhost or Docker
      -> backend app on localhost or Docker

• Nginx handles public ports 80 and 443.
• Nginx can handle HTTPS certificates.
• Nginx routes domains and paths to the correct app.
• Nginx keeps app ports hidden from the public internet.
• Nginx can reload config without fully restarting the service.

AWS ALB and Nginx

• AWS Application Load Balancer can replace Nginx as the public reverse proxy in many AWS-native setups.
• ALB is useful with multiple EC2 instances, ECS services, target groups, health checks, and autoscaling.
• Nginx is often simpler for a single EC2 instance.

Single EC2 pattern:

Browser
  -> Nginx on EC2
    -> app services or Docker containers

AWS ALB pattern:

Browser
  -> AWS Application Load Balancer
    -> EC2 instance
    -> ECS task
    -> target group

Process and PID inspection

ps aux
ps -p 1 -o pid,comm,args

ps aux | grep nginx
ps aux | grep docker
ps aux | grep node

ps aux | grep '[n]ginx'

pgrep nginx
pgrep -a nginx

pstree -p

top
htop

Install helper tools on Arch if needed.

sudo pacman -S psmisc htop

Kernel threads in ps output

Processes shown inside square brackets are usually kernel threads, not normal user services.

root           1  0.0  0.0  26452 15516 ? Ss Apr30 0:03 /sbin/init
root           2  0.0  0.0      0     0 ? S  Apr30 0:00 [kthreadd]
root           3  0.0  0.0      0     0 ? S  Apr30 0:00 [pool_workqueue]
root          15  0.0  0.0      0     0 ? S  Apr30 0:01 [ksoftirqd/0]

• PID 1 is usually systemd or /sbin/init.
• PID 2 is kthreadd, the kernel thread manager.
• kworker entries are kernel worker threads.
• ksoftirqd entries are kernel interrupt handling workers.

Check listening ports

Use ss to see which processes listen on ports. Add sudo to see process names clearly.

ss -ltnp
sudo ss -ltnp

sudo ss -ltnp | grep 3000
sudo ss -ltnp | grep 8000
sudo ss -ltnp | grep 80
sudo ss -ltnp | grep 443

journalctl and systemd-journald

• systemd-journald is the background log collection daemon.
• journalctl is the command used to read logs.
• Use journalctl for services managed directly by systemd.

journalctl -u nginx
journalctl -u docker
journalctl -u my-app.service

journalctl -u my-app.service -f
journalctl -u my-app.service -n 100
journalctl -u my-app.service --since "1 hour ago"
journalctl -u my-app.service --since today

journalctl -b
journalctl -u my-app.service -b

Check journald itself:

systemctl status systemd-journald
ps aux | grep '[s]ystemd-journald'

Docker logs and journal logs

• journalctl -u docker shows Docker daemon logs.
• docker compose logs shows logs from application containers.
• Use Docker logs for app exceptions inside containers.
• Use journalctl -u docker for Docker engine, network, image, and daemon problems.

systemd
  -> docker.service
    -> Docker daemon
      -> containers
        -> backend app
        -> frontend app

journalctl -u docker -f

docker compose logs -f backend
docker compose logs -f frontend

docker compose logs --tail=100 backend
docker compose logs --tail=100 frontend

Cron and cronie

• cron is a time-based command scheduler.
• cronie is a common cron implementation on Arch.
• crontab edits the current user’s scheduled jobs.
• Cron uses the system’s local timezone by default.

sudo pacman -S cronie
sudo systemctl enable --now cronie
systemctl status cronie

crontab -e
crontab -l

Use nano if vi is missing.

EDITOR=nano crontab -e

echo 'export EDITOR=nano' >> ~/.bashrc
source ~/.bashrc

Cron time format

minute hour day-of-month month day-of-week command

Example: run a Python script every day at 03:00.

0 3 * * * /home/example/project/.venv/bin/python /home/example/project/scripts/daily_job.py

Check local system time and timezone.

date
timedatectl
timedatectl | grep "Time zone"

Set timezone if needed.

sudo timedatectl set-timezone Europe/Istanbul

Cron logging pattern

Redirect command output to a log file when running scripts from cron. This keeps print output and errors easy to inspect.

mkdir -p /home/example/project/logs

0 3 * * * cd /home/example/project && /home/example/project/.venv/bin/python scripts/daily_job.py >> logs/daily_job.log 2>&1

• appends normal output to the log file.

• 2>&1 sends error output to the same log file.

tail -f /home/example/project/logs/daily_job.log

journalctl -u cronie --since today

Cron versus Celery

Use case	Better tool
Run one Python script every day	cron
Run backup or cleanup commands	cron
Send verification email after user registration	Celery
Process uploads or reports in the background	Celery
Retry failed external API work	Celery
Schedule app-level tasks with workers	Celery beat

cron
  At this time, run this shell command.

Celery
  Put this app task into a background queue.

Celery beat
  At this time, put this app task into the Celery queue.

Redis
  Broker where Celery tasks wait.

Celery worker
  Process that consumes queued tasks.

For web apps, slow or retryable work should usually move out of the request and into a background worker.

User registers
  -> backend creates user
  -> backend queues send_verification_email task
  -> backend returns response quickly

Celery worker
  -> sends email separately
  -> retries if needed

Quick service cleanup checklist

sudo systemctl stop example.service
sudo systemctl disable example.service
sudo systemctl cat example.service
sudo rm /etc/systemd/system/example.service
sudo systemctl daemon-reload
sudo systemctl reset-failed
systemctl status example.service

Expected final output:

Unit example.service could not be found.

Quick Nginx cleanup checklist

ls -l /etc/nginx/sites-enabled | grep example

sudo rm -f /etc/nginx/sites-enabled/example.conf
sudo rm -f /etc/nginx/sites-available/example.conf

sudo nginx -t
sudo systemctl reload nginx

ls /etc/nginx/sites-available
ls -l /etc/nginx/sites-enabled

Debug command set

systemctl status nginx
systemctl status docker
systemctl list-unit-files | grep example
systemctl list-units --all | grep example

journalctl -u nginx -n 100
journalctl -u docker -n 100

docker compose logs --tail=100 backend
docker compose logs --tail=100 frontend

ps -p 1 -o pid,comm,args
ps aux | grep '[n]ginx'
pgrep -a nginx

sudo ss -ltnp
sudo ss -ltnp | grep 8000

Reference paths

• Custom systemd units: /etc/systemd/system/
• Packaged systemd units on Arch: /usr/lib/systemd/system/
• Nginx main config: /etc/nginx/nginx.conf
• Nginx available sites: /etc/nginx/sites-available/
• Nginx enabled sites: /etc/nginx/sites-enabled/
• User crontab editor: crontab -e
• Docker Compose logs: docker compose logs
• Systemd service logs: journalctl -u service-name