Stage 7 · Master

Python for Operators

Files, Paths & OS

pathlib, os, shutil — read configs, walk directories, and manipulate files.

7 min readPython for DevOpsMaster

pathlib Over os.path

Python 3.4+ includes pathlib, an object-oriented interface for filesystem paths. It replaces string manipulation with method chaining. Always prefer pathlib over os.path — it is more readable and less error-prone.

pathlib basics


from pathlib import Path

# Construct paths — OS-agnostic, no string concatenation
config_dir = Path("/etc/myapp")
log_dir = Path("/var/log/myapp")

# Join paths with / operator
config_file = config_dir / "settings.yaml"
log_file = log_dir / "app.log"

# Resolve relative paths
project_root = Path(".")
absolute = (project_root / "src" / "main.py").resolve()

# Check existence
if config_file.exists():
    print(f"Config found: {config_file}")

The / operator joins path segments correctly on any OS. You never need to worry about forward slashes vs backslashes.

Reading Files

pathlib makes file reading clean. Always specify encoding explicitly — never rely on the system default.

Read entire file and read line by line


from pathlib import Path

# Read entire file
config_path = Path("/etc/myapp/settings.yaml")
content = config_path.read_text(encoding="utf-8")

# Read as bytes (for binary files)
binary_data = config_path.read_bytes()

# Read line by line (memory efficient for large files)
log_path = Path("/var/log/app.log")
for line in log_path.read_text(encoding="utf-8").splitlines():
    if "ERROR" in line:
        print(line)

read_text() loads the entire file into memory. For large log files, read line by line instead of loading the whole file.

Always specify encoding

read_text() without encoding= uses the system default, which varies between machines. Always pass encoding='utf-8' explicitly to avoid UnicodeDecodeError on different systems.

Writing Files

Write, append, and atomic writes


from pathlib import Path

# Write (overwrites existing)
output = Path("/tmp/report.txt")
output.write_text("Line 1
Line 2
", encoding="utf-8")

# Append
with open(output, "a", encoding="utf-8") as f:
    f.write("Line 3
")

# Atomic write — write to temp, then rename
# Prevents partial writes if the process crashes mid-write
def atomic_write(target: Path, content: str) -> None:
    tmp = target.with_suffix(".tmp")
    tmp.write_text(content, encoding="utf-8")
    tmp.rename(target)  # Atomic on same filesystem

atomic_write(Path("/etc/app/config.yaml"), "key: value
")

Atomic writes are critical for config files. If your script crashes while writing, the original file remains intact.

Directory Operations

Create, list, walk, and glob


from pathlib import Path

# Create directories (parents=True creates intermediate dirs)
output_dir = Path("/tmp/reports/2024/january")
output_dir.mkdir(parents=True, exist_ok=True)

# List directory contents
for item in Path(".").iterdir():
    if item.is_file():
        print(f"File: {item.name}")
    elif item.is_dir():
        print(f"Dir:  {item.name}")

# Glob — find files matching patterns
for log in Path("/var/log").glob("**/*.log"):
    print(log)

# Walk — iterate with directory depth
for dirpath, dirnames, filenames in Path(".").walk():
    level = len(dirpath.parts)
    indent = "  " * level
    print(f"{indent}{dirpath.name}/")

glob('**/*.log') matches recursively. The double star means 'any number of intermediate directories'.

File Metadata

Stat, permissions, and modification time


from pathlib import Path
import time
from datetime import datetime

file = Path("deploy.sh")

# Get metadata
stat = file.stat()
print(f"Size: {stat.st_size} bytes")
print(f"Modified: {datetime.fromtimestamp(stat.st_mtime)}")

# Check permissions
import os
if os.access(file, os.X_OK):
    print("File is executable")
else:
    os.chmod(file, 0o755)

# Check if path is a file, directory, or symlink
p = Path("/usr/bin/python3")
print(p.is_file())      # True
print(p.is_dir())       # False
print(p.is_symlink())   # True
print(p.resolve())      # Follow symlink to real path

stat() returns a os.stat_result object with size, timestamps, and permissions. Use datetime.fromtimestamp() to convert Unix timestamps to readable dates.

shutil — High-Level Ops

shutil provides high-level file operations that pathlib does not cover — copying, moving, and deleting directory trees.

Copy, move, and archive with shutil


import shutil
from pathlib import Path

# Copy a file
src = Path("config.yaml")
shutil.copy2(src, "/backup/config.yaml")  # preserves metadata

# Copy an entire directory tree
shutil.copytree("src/", "src_backup/")

# Move / rename
shutil.move("/tmp/deploy.sh", "/usr/local/bin/deploy.sh")

# Remove entire directory tree
shutil.rmtree("/tmp/build_output/")

# Create a tar.gz archive
shutil.make_archive("release", "gztar", "dist/")

# Extract an archive
shutil.unpack_archive("release.tar.gz", "extracted/")

copy2 preserves file metadata (timestamps, permissions). copytree recursively copies everything. rmtree deletes everything — use with caution.

Use shutil.disk_usage for capacity checks

shutil.disk_usage('/') returns total, used, and free bytes. Useful for scripts that need to check disk space before writing large files.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.