Operational View
Describes how the system is operated, monitored, and maintained once deployed. Covers health checking, error handling, configuration management, and known operational gaps.
System Health Checks
Container orchestration (Docker Compose in dev, Kubernetes in production) continuously polls each service’s dedicated health check. The API’s /health endpoint is the top-level signal for the application layer. Each infrastructure service exposes its own probe — PostgreSQL via pg_isready, Redis via redis-cli ping, MQTT via mosquitto_sub, and Keycloak via its /health/ready endpoint. All five must report healthy for the platform to be considered fully operational; a failure in any one of them will block dependent service startup.
flowchart TD
subgraph HealthChecks["Health Check Endpoints"]
HC["GET /health\n(ASP.NET Core built-in)"]
PGH["pg_isready -U postgres\n(postgres-dev)"]
RH["redis-cli ping\n(redis-dev)"]
MQH["mosquitto_sub -t $$SYS/broker/uptime\n(mqtt-dev)"]
KCH["curl /health/ready\n(keycloak-dev)"]
end
ORCH["Container Orchestrator\n(Docker Compose / Kubernetes)"] -->|polls| HC
ORCH --> PGH
ORCH --> RH
ORCH --> MQH
ORCH --> KCH
Sub-domain Mapping — Health Checks
| Health Check |
Business Sub-domain |
What breaks if it fails |
GET /health (API) |
All sub-domains |
Entire API unavailable — no bouquet discovery, orders, or vendor management |
pg_isready (PostgreSQL) |
All sub-domains |
No persistence; all commands and queries fail immediately |
redis-cli ping (Redis) |
Identity & Tenancy, Customer Purchasing, Payments |
Token blacklist, idempotency, and query cache all unavailable; duplicate orders possible |
mosquitto_sub (MQTT) |
IoT / MQTT |
Vase telemetry (heartbeats, sensor data) not ingested; freshness scores stale |
/health/ready (Keycloak) |
Identity & Tenancy |
Vendor and customer login fails; dev-token mode unaffected |
Request Error Handling Pipeline
flowchart TD
REQ["Incoming HTTP Request"] --> GEX["GlobalExceptionMiddleware\n(catches all unhandled exceptions)"]
GEX -->|unhandled exception| PD["ProblemDetails response\n500 Internal Server Error"]
GEX --> AUTH["Authentication\n(401 Unauthorized if JWT invalid/missing)"]
AUTH --> TENANT["TenantResolutionMiddleware\n(403 if claims missing for protected routes)"]
TENANT --> BLACKLIST["TokenBlacklistMiddleware\n(401 if JTI revoked in Redis)"]
BLACKLIST --> AUTHZ["Authorization Policies\n(403 if role/policy mismatch)"]
AUTHZ --> RATE["RateLimiter\n(429 Too Many Requests)"]
RATE --> IDEMP["IdempotencyMiddleware\n(replays cached response if key exists)"]
IDEMP --> CTRL["Controller → MediatR Handler"]
CTRL -->|Result.Error| MAP["Error string → HTTP status mapping\n(400 / 404 / 409 / 422 etc.)"]
MAP --> RESP["HTTP Response"]
Sub-domain Mapping — Error Pipeline
| Middleware layer |
Business Sub-domain |
Error it handles |
GlobalExceptionMiddleware |
All sub-domains |
Uncaught exceptions — wraps as 500 ProblemDetails |
| Authentication (401) |
Identity & Tenancy |
Missing or invalid Keycloak/dev JWT token |
TenantResolutionMiddleware (403) |
Identity & Tenancy |
JWT present but lacks required vendor_id / customer_id claims |
TokenBlacklistMiddleware (401) |
Identity & Tenancy |
Revoked JWT jti — logged-out session attempting reuse |
| Authorization Policies (403) |
All sub-domains |
Role/policy mismatch (e.g., Customer accessing a Vendor endpoint) |
RateLimiter (429) |
All sub-domains |
Abuse protection; applies globally across all routes |
IdempotencyMiddleware (replays) |
Customer Purchasing / Payments |
Duplicate POST /orders or payment confirmation with same idempotency key |
| Controller error mapping |
All sub-domains |
Result.Error string prefixes (bouquet-unavailable, vendor-mismatch, …) mapped to 400 / 404 / 409 |
Logging
The API uses ASP.NET Core’s built-in structured logging with RequestIdMiddleware injecting X-Request-ID on every request, enabling log correlation across services.
| Log Level |
Logged By |
Information |
Request start/end (LoggingBehaviour), domain event dispatch |
Warning |
Validation failures, cache misses, token blacklist hits |
Error |
Unhandled exceptions (GlobalExceptionMiddleware), Keycloak admin API failures |
Log files are written to src/backend/FlowerShop.API/logs/ (mapped via Docker volume api_dev_logs).
Operational Runbook — Common Procedures
# Start infrastructure
docker-compose -f docker/docker-compose.dev.yml up -d
# Start the API (hot reload)
cd src/backend/FlowerShop.API
dotnet watch run
# Start portals (separate terminals)
cd src/portals/FlowerShop.AdminPortal && dotnet run # :5001
cd src/portals/FlowerShop.VendorPortal && dotnet run # :5002
cd src/portals/FlowerShop.CustomerApp && dotnet run # :5003
Running database migrations
cd src/backend/FlowerShop.Infrastructure
dotnet ef migrations add <MigrationName> --startup-project ../FlowerShop.API
cd ../FlowerShop.API
dotnet ef database update
Running tests
dotnet test # runs all 126 tests across 3 test projects
Checking Swagger
Navigate to http://localhost:8080/swagger — all 23 controllers and their endpoints are documented.
Sub-domain Mapping — Monitoring Signals
| Signal |
Business Sub-domain |
Meaning |
/health → unhealthy |
All sub-domains |
DB or Redis unreachable — immediate action required |
outbox_messages accumulating |
All sub-domains |
OutboxDispatcherService stalled or InMemoryEventBus failing — domain events not delivered |
SmartVase.ConnectionStatus = Offline spike |
IoT / MQTT |
MQTT broker issue or network partition affecting vases; freshness scoring degraded |
| Redis memory growing |
Identity & Tenancy, Customer Purchasing |
Token blacklist or idempotency key TTL misconfigured; check key expiry policies |
| Keycloak admin API errors |
Identity & Tenancy |
Keycloak unreachable; vendor onboarding (VendorOnboardingService) will fail silently |
Known Operational Gaps (active stubs)
| Gap |
Impact |
Planned Fix |
RabbitMQEventBus.PublishAsync returns Task.CompletedTask |
Domain events are silently dropped; InMemoryEventBus is active instead |
EP-09 S-09-06 |
AnalyticsController calls VendorId.New() in 4 actions |
All analytics data is wrong (random GUID instead of tenant’s VendorId) |
EP-10 T-10-001 |
VendorPortal reads VendorId from session instead of JWT claim |
Vendor isolation broken for portal users |
EP-12 T-12-003 |
VendorPortal POSTs photos to hardcoded http://localhost:8080/api/mock-vase/... |
Photo upload fails in any non-localhost environment |
EP-12 T-12-004 |
INotificationServiceClient is a stub |
FCM push notifications not delivered to mobile devices |
EP-06 |
IStripeService is a stub |
Payment not wired in CreateOrderCommandHandler |
EP-04 |
| Cart + OrderLine DB tables do not exist |
Cart and multi-line order endpoints return DB errors until migration is run |
Run AddCartAndOrderLines migration |
CachingGeocodingService uses IMemoryCache |
Geocoding cache is process-local; lost on restart; not shared across instances |
Intentional for now; revisit in EP-09 |
Monitoring Signals
| Signal |
Source |
Meaning |
/health → unhealthy |
API process |
Immediate attention — DB or Redis unreachable |
outbox_messages rows with processed_at IS NULL accumulating |
PostgreSQL |
OutboxDispatcherService not running or event bus failing |
SmartVase.ConnectionStatus = Offline spike |
PostgreSQL |
MQTT broker issue or network partition affecting vases |
| Redis memory usage growing |
Redis |
Token blacklist or idempotency key TTL misconfigured |
| Keycloak admin API errors in logs |
API logs |
Keycloak service unreachable; vendor onboarding will fail |
Sub-domain Mapping — Operational Gaps
| Gap |
Business Sub-domain |
Impact |
RabbitMQEventBus stub |
All sub-domains (async events) |
Domain events dropped in production; InMemoryEventBus is the active fallback |
AnalyticsController calls VendorId.New() |
Vendor Management / Platform Administration |
All analytics responses return data for a random, non-existent vendor |
VendorPortal reads VendorId from session |
Vendor Management |
Vendor isolation broken in the portal — any session can access any vendor’s data |
INotificationServiceClient stub |
Customer Purchasing |
FCM push notifications not delivered to any mobile device |
IStripeService stub |
Payments |
Orders complete without actual payment processing |
| Cart + OrderLine tables missing |
Customer Purchasing |
Cart and multi-line order endpoints return DB errors until migration is run |
Security Operations
flowchart LR
subgraph TokenLifecycle["JWT Token Lifecycle"]
ISSUE["Keycloak issues JWT\n(sub, role, vendor_id, vendor_type, customer_id)"]
USE["API validates JWT\nTenantResolutionMiddleware reads claims"]
REVOKE["POST /auth/revoke\nJTI added to Redis blacklist"]
CHECK["TokenBlacklistMiddleware\nchecks Redis on every request"]
end
ISSUE --> USE --> REVOKE --> CHECK --> USE
- Token revocation is synchronous and Redis-backed — revoked tokens are blocked on the next request.
- Multi-tenancy isolation is enforced at the MediatR layer via
TenantValidationBehavior — handlers cannot accidentally access another tenant’s data.
- Rate limiting is applied globally via ASP.NET Core’s built-in
RateLimiter middleware.
- Idempotency prevents duplicate orders even under concurrent client retries.