Stage 7 · Master
Automation Scripts
Config File Management
Parse YAML, JSON, TOML, INI — validate, transform, and regenerate configs.
Config Format Overview
| Format | Best For | Python Library | Extension |
|---|---|---|---|
| YAML | Kubernetes, Ansible, Docker Compose | pyyaml, ruamel.yaml | .yaml/.yml |
| JSON | APIs, CI pipelines | json (stdlib) | .json |
| TOML | pyproject.toml, Cargo.toml | tomli, tomllib (3.11+) | .toml |
| INI | Legacy config, .cfg files | configparser (stdlib) | .ini/.cfg |
Do not use pyyaml for anything you care about. ruamel.yaml preserves comments and is safer for configs that humans edit. For Python 3.11+, use tomllib from the standard library for TOML.
YAML Parsing
from ruamel.yaml import YAML
yaml = YAML()
yaml.preserve_quotes = True
# Load YAML
config = yaml.load(Path("config.yaml"))
# Access nested values
database = config["services"]["database"]
host = database["host"]
port = database.get("port", 5432) # Default if missing
# Modify and preserve comments
config["services"]["database"]["port"] = 5433
# Dump back to file (preserves comments and formatting)
yaml.dump(config, Path("config.yaml"))ruamel.yaml preserves comments, quotes, and formatting. pyyaml strips all comments and can execute arbitrary Python objects with yaml.load().
pyyaml's yaml.load() without a Loader argument can execute arbitrary Python code from YAML files. Always use yaml.safe_load() or switch to ruamel.yaml.
JSON Config
import json
from pathlib import Path
def load_json_config(path: Path) -> dict:
"""Load a JSON config file with comments stripped."""
content = path.read_text(encoding="utf-8")
# Strip single-line comments (JSON does not support them natively)
lines = []
for line in content.splitlines():
stripped = line.strip()
if stripped.startswith("//"):
continue
lines.append(line)
return json.loads("\n".join(lines))
# Pretty-print JSON configs
def write_json_config(data: dict, path: Path) -> None:
path.write_text(
json.dumps(data, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
# Merge two JSON configs (second overrides first)
def merge_configs(base: dict, override: dict) -> dict:
result = base.copy()
for key, value in override.items():
if key in result and isinstance(result[key], dict) and isinstance(value, dict):
result[key] = merge_configs(result[key], value)
else:
result[key] = value
return resultJSON does not support comments. Some tools allow // comments, but they are not standard. Use YAML or TOML if comments are important.
TOML Config
import tomllib
from pathlib import Path
# Python 3.11+ has tomllib in stdlib
# For older versions: pip install tomli
def load_toml_config(path: Path) -> dict:
with open(path, "rb") as f:
return tomllib.load(f)
# Write TOML (tomllib is read-only; use tomli_w or tomli-w for writing)
# pip install tomli-w
import tomli_w
def write_toml_config(data: dict, path: Path) -> None:
with open(path, "wb") as f:
tomli_w.dump(data, f)
# Example: read pyproject.toml
config = load_toml_config(Path("pyproject.toml"))
project_name = config["project"]["name"]
dependencies = config["project"]["dependencies"]tomllib is in Python 3.11+ standard library. It is read-only. For writing TOML, install tomli-w. TOML is the best format for configs that need both human readability and type safety.
Config Validation
from pydantic import BaseModel, Field, field_validator
from typing import Optional
from pathlib import Path
import tomllib
class DatabaseConfig(BaseModel):
host: str
port: int = Field(default=5432, ge=1, le=65535)
username: str
password: str
ssl: bool = True
@field_validator("host")
@classmethod
def host_must_be_valid(cls, v: str) -> str:
if not v or v.startswith("-"):
raise ValueError(f"Invalid host: {v}")
return v
class AppConfig(BaseModel):
app_name: str
debug: bool = False
database: DatabaseConfig
log_level: str = "INFO"
def load_and_validate(path: Path) -> AppConfig:
with open(path, "rb") as f:
raw = tomllib.load(f)
return AppConfig(**raw)
# Usage — raises ValidationError with clear messages
config = load_and_validate(Path("config.toml"))Pydantic validates types, ranges, and custom constraints automatically. The error messages tell you exactly what is wrong with the config.
Config Transformation
from ruamel.yaml import YAML
import tomli_w
import json
from pathlib import Path
# YAML to TOML
yaml = YAML()
with open("config.yaml") as f:
data = yaml.load(f)
with open("config.toml", "wb") as f:
tomli_w.dump(data, f)
# JSON to YAML
with open("config.json") as f:
data = json.load(f)
yaml.dump(data, Path("config.yaml"))
# Template config with environment variables
import os
def template_config(template: str, env: dict[str, str] | None = None) -> str:
"""Replace ${VAR} placeholders with environment values."""
env = env or dict(os.environ)
for key, value in env.items():
template = template.replace(f"${{{key}}}", value)
return template
template = Path("config.yaml.tmpl").read_text()
result = template_config(template)
Path("config.yaml").write_text(result)Config transformation is common when migrating between tools (Ansible to Terraform, Docker Compose to Kubernetes). Build small scripts to automate the conversion.
Never store secrets in config files. Use environment variables or a secrets manager (Vault, AWS SSM, Azure Key Vault). Config files should contain only non-sensitive settings.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.