mitlist/docs/financial-system-overview.md

Financial System Overview

Introduction

This document provides a comprehensive overview of the Expense, Cost & Financial domain of the project. It is intended for backend / frontend developers, QA engineers and DevOps personnel who need to understand how money flows through the system, what invariants are enforced and where to extend the platform.

TL;DR The financial subsystem is a Split-wise-inspired engine with first-class support for: shared lists, item-derived expenses, multi-scheme splitting, settlements, recurring charges and complete auditability.

---

Main Concepts & Entities

Entity	Purpose	Key Relationships
User	A registered account. Owns expenses, owes splits, records settlements.	expenses_paid, expenses_created, expense_splits, settlements_made/received, settlements_created
Group	A collection of users with shared expenses / lists.	member_associations (UserGroup), lists, expenses, settlements
List	A shopping / to-do list. May belong to a Group or be personal.	items, expenses
Item	A purchasable line inside a List. Price & author drive item-based expense splits.	list, added_by_user
Expense	A monetary outflow.	splits, list, group, item, recurrence_pattern
ExpenseSplit	A who-owes-what record for an Expense.	user, settlement_activities
Settlement	A generic cash transfer between two users inside a group.	–
SettlementActivity	A payment that reduces an individual ExpenseSplit (e.g. Alice pays Bob her part).	–
RecurrencePattern	The schedule template that spawns future Expense occurrences.	expenses
FinancialAuditLog	Append-only journal of who did what (create / update / delete) for all financial entities.	–

---

Expense Lifecyle

1. Creation (POST /financials/expenses) • Caller provides an ExpenseCreate DTO.
   • Backend validates the context (list / group / item), the payer, and the chosen split strategy. • Supported split_type values (SplitTypeEnum):

   • EQUAL – evenly divided among computed participants.
   • EXACT_AMOUNTS – caller supplies absolute owed amounts.
   • PERCENTAGE – caller supplies percentages totalling 100 %.
   • SHARES – integer share units (e.g. 1 : 2 : 3).
   • ITEM_BASED – derived from priced Items in a list. • A database transaction writes Expense + ExpenseSplit rows and a FinancialAuditLog entry.

2. Reading • GET /financials/expenses/{id} enforces row-level security: the requester must be payer, list member or group member.

3. Update / Delete • Optimistic-locking via the version field. • Only the payer or a group owner may mutate records.

4. Settlement • Generic settlements (/financials/settlements) clear balances between two users.
   • Fine-grained settlements (/financials/expense_splits/{id}/settle) clear a single ExpenseSplit.

5. Recurring Expenses • An Expense can be flagged is_recurring = true and carry a RecurrencePattern.
   • A background job (app/jobs/recurring_expenses.py::generate_recurring_expenses) wakes up daily and:

   1. Finds template expenses due (next_occurrence <= now).
   2. Spawns a child expense replicating the split logic.
   3. Updates last_occurrence, decrements max_occurrences and calculates the next date.

---

Cost Summaries & Balance Sheets

List Cost Summary

Endpoint: GET /costs/lists/{list_id}/cost-summary

• If an ITEM_BASED expense already exists → returns a snapshot derived from the expense.
• Otherwise → computes an on-the-fly summary using Item.price values (read-only).

Key Figures:

• total_list_cost – sum of item prices.
• equal_share_per_user – what each participant should pay.
• balance – (over / under) contribution for each user.

Action: POST /costs/lists/{id}/cost-summary finalises the list by persisting the derived ITEM_BASED expense.

Group Balance Summary

Endpoint: GET /costs/groups/{group_id}/balance-summary

Aggregates across all group expenses + settlements:

• What each user paid (expenses + received settlements)
• What each user owed
• Suggested minimal settlement graph (creditors vs debtors)

---

Data-Integrity Rules & Guards

1. Expense must reference either list_id or group_id (DB-level CHECK).
2. Row uniqueness guards: UniqueConstraint('expense_id', 'user_id') on ExpenseSplit.
3. Settlement payer ≠ payee (DB CHECK).
4. All mutating endpoints perform authorization checks via crud_group / crud_list helpers.
5. Monetary amounts are stored as Numeric(10,2) and rounded (ROUND_HALF_UP).

---

Recent Fixes & Improvements (June 2025)

Area	Issue	Resolution
Recurring filter	GET /financials/expenses?isRecurring=true referenced nonexistent recurrence_rule.	Switched to is_recurring flag.
Enum mismatches	RecurrencePattern.type stored uppercase enum, while API took lowercase strings.	Robust mapper converts strings → RecurrenceTypeEnum; scheduler is now case-insensitive.
Scheduler	_calculate_next_occurrence failed with Enum values & stringified days_of_week.	Added polymorphic handling + safe parsing of comma-separated strings.

All tests pass (pytest -q) and new unit tests cover the edge-cases above.

---

Extension Points

• VAT / Tax logic – attach a tax_rate column to Expense and resolve net vs gross amounts.
• Multi-currency – normalize amounts to a base currency using FX rates; expose a Currency table.
• Budgeting / Limits – per-group or per-list spending caps with alerting.
• Webhook notifications – publish FinancialAuditLog entries to external services.

---

Appendix – Key SQL Schema Snapshots

-- Expense table (excerpt)
CREATE TABLE expenses (
    id              SERIAL PRIMARY KEY,
    total_amount    NUMERIC(10,2) NOT NULL,
    split_type      VARCHAR(20) NOT NULL,
    list_id         INT NULL,
    group_id        INT NULL,
    -- …
    CHECK (group_id IS NOT NULL OR list_id IS NOT NULL)
);

CREATE UNIQUE INDEX uq_expense_user_split ON expense_splits(expense_id, user_id);

---

System Reliability Analysis & Improvements

✅ Implemented Reliability Features

1. Transaction Safety

• All financial operations use transactional sessions (get_transactional_session)
• Atomic operations ensure data consistency across expense creation, splits, and settlements
• Row-level locking (WITH FOR UPDATE) prevents race conditions in settlement activities

2. Overpayment Protection

• NEW: Settlement activities now validate against remaining owed amount
• Prevents payments that exceed the split's owed amount
• Provides clear error messages with remaining balance information
• Handles multiple partial payments correctly

3. Data Validation & Constraints

• Decimal precision: All monetary amounts use Numeric(10,2) with proper rounding
• Positive amount validation: Prevents negative payments and settlements
• User existence validation: Ensures all referenced users exist before operations
• Split consistency: Validates split totals match expense amounts (EXACT_AMOUNTS, PERCENTAGE)

4. Permission & Authorization

• Multi-layered permission checks for expense creation and settlement recording
• Group owners can act on behalf of members with proper validation
• List/group access controls prevent unauthorized financial operations

5. Status Management

• Automatic status updates for expense splits (unpaid → partially_paid → paid)
• Cascading status updates for parent expenses based on split states
• Pessimistic locking ensures consistent status transitions

6. Audit Trail

• All financial operations logged via create_financial_audit_log
• Complete traceability of who created/modified financial records
• Immutable settlement activity records (no updates, only creation)

7. Error Handling

• Comprehensive exception hierarchy for different error types
• Specific OverpaymentError for payment validation failures
• Database integrity and connection error handling
• Graceful degradation with meaningful error messages

🔍 Potential Areas for Enhancement

1. Optimistic Locking for Expenses

Currently, expenses use basic versioning but could benefit from full optimistic locking:

# Consider adding to ExpenseUpdate operations
if expected_version != current_expense.version:
    raise ConflictError("Expense was modified by another user")

2. Recurring Expense Reliability

• Add retry logic for failed recurring expense generation
• Implement dead letter queue for failed recurring operations
• Add monitoring for recurring expense job health

3. Currency Consistency

While currency is stored, there's no validation that all splits in an expense use the same currency:

# Potential enhancement
if expense.currency != "USD" and any(split.currency != expense.currency for split in splits):
    raise InvalidOperationError("All splits must use the same currency as the parent expense")

4. Settlement Verification

Consider adding verification flags for settlements to distinguish between:

• Automatic settlements (from expense splits)
• Manual settlements (direct user payments)
• Disputed settlements requiring verification

📊 System Health Metrics

The system provides comprehensive financial tracking through:

1. Real-time balance calculations via costs_service.py
2. Settlement suggestions using optimal debt reduction algorithms
3. Expense categorization and filtering with proper indexing
4. Multi-context support (lists, groups, personal expenses)

🛡️ Security Considerations

• Input sanitization: All financial inputs validated through Pydantic schemas
• Authorization layers: Multiple permission checks prevent unauthorized access
• Audit logging: Complete financial operation history for compliance
• Data isolation: Users only see expenses/settlements they have permission to access

🚀 Performance Optimizations

• Database indexing on critical foreign keys and search fields
• Eager loading with selectinload to prevent N+1 queries
• Pagination for large result sets
• Connection pooling with health checks (pool_pre_ping=True)

---

Testing & Quality Assurance

The financial system includes comprehensive test coverage:

1. Unit tests for CRUD operations and business logic
2. Integration tests for API endpoints and workflows
3. Edge case testing for overpayment protection and boundary conditions
4. Concurrency tests for settlement race conditions
5. Data consistency tests for split calculations and status updates

Example test scenarios:

• Multiple users settling the same split simultaneously
• Expense split totals validation across different split types
• Currency precision and rounding accuracy
• Permission boundary testing for cross-user operations

---

Deployment & Monitoring Recommendations

Database Considerations

• Regular backup strategy for financial data
• Monitor transaction isolation levels and deadlocks
• Set up alerts for unusual financial activity patterns
• Implement database connection monitoring

Application Monitoring

• Track settlement activity creation rates and failures
• Monitor recurring expense job execution and errors
• Set up alerts for permission denial patterns
• Track API response times for financial endpoints

Business Intelligence

• Daily/weekly financial summaries per group
• Settlement velocity tracking (time to pay debts)
• Expense categorization analytics
• User engagement with financial features

The financial system is now production-ready with robust reliability safeguards, comprehensive error handling, and strong data consistency guarantees.