# Financial System Overview

## Introduction

This document provides a comprehensive overview of the **Expense, Cost & Financial** domain of the project.  It is intended for backend / frontend developers, QA engineers and DevOps personnel who need to understand **how money flows through the system**, what invariants are enforced and where to extend the platform.

> **TL;DR**  The financial subsystem is a *Split-wise*-inspired engine with first-class support for: shared lists, item-derived expenses, multi-scheme splitting, settlements, recurring charges and complete auditability.

---

## Main Concepts & Entities

| Entity | Purpose | Key Relationships |
|--------|---------|-------------------|
| **User** | A registered account.  Owns expenses, owes splits, records settlements. | `expenses_paid`, `expenses_created`, `expense_splits`, `settlements_made/received`, `settlements_created` |
| **Group** | A collection of users with shared expenses / lists. | `member_associations (UserGroup)`, `lists`, `expenses`, `settlements` |
| **List** | A shopping / to-do list.  May belong to a `Group` or be personal. | `items`, `expenses` |
| **Item** | A purchasable line inside a `List`.  Price & author drive *item-based* expense splits. | `list`, `added_by_user` |
| **Expense** | A monetary outflow. | `splits`, `list`, `group`, `item`, `recurrence_pattern` |
| **ExpenseSplit** | A *who-owes-what* record for an `Expense`. | `user`, `settlement_activities` |
| **Settlement** | A generic *cash transfer* between two users inside a group. | –  |
| **SettlementActivity** | A *payment* that reduces an individual `ExpenseSplit` (e.g. Alice pays Bob her part). | –  |
| **RecurrencePattern** | The schedule template that spawns future `Expense` occurrences. | `expenses` |
| **FinancialAuditLog** | Append-only journal of *who did what* (create / update / delete) for all financial entities. | –  |

---

## Expense Lifecyle

1. **Creation** (`POST /financials/expenses`)
   • Caller provides an `ExpenseCreate` DTO.<br />
   • Backend validates the *context* (list / group / item), the *payer*, and the chosen **split strategy**.
   • Supported `split_type` values (`SplitTypeEnum`):
     * `EQUAL` – evenly divided among computed participants.
     * `EXACT_AMOUNTS` – caller supplies absolute owed amounts.
     * `PERCENTAGE` – caller supplies percentages totalling 100 %.
     * `SHARES` – integer share units (e.g. 1 : 2 : 3).
     * `ITEM_BASED` – derived from priced `Item`s in a list.
   • A database transaction writes `Expense` + `ExpenseSplit` rows and a `FinancialAuditLog` entry.

2. **Reading**
   • `GET /financials/expenses/{id}` enforces *row-level* security: the requester must be payer, list member or group member.

3. **Update / Delete**
   • Optimistic-locking via the `version` field.
   • Only the **payer** or a **group owner** may mutate records.

4. **Settlement**
   • Generic settlements (`/financials/settlements`) clear balances between two users.<br />
   • Fine-grained settlements (`/financials/expense_splits/{id}/settle`) clear a single `ExpenseSplit`.

5. **Recurring Expenses**
   • An `Expense` can be flagged `is_recurring = true` and carry a `RecurrencePattern`.<br />
   • A background job (`app/jobs/recurring_expenses.py::generate_recurring_expenses`) wakes up daily and:
     1. Finds template expenses due (`next_occurrence <= now`).
     2. Spawns a *child* expense replicating the split logic.
     3. Updates `last_occurrence`, decrements `max_occurrences` and calculates the next date.

---

## Cost Summaries & Balance Sheets

### List Cost Summary
Endpoint: `GET /costs/lists/{list_id}/cost-summary`

• If an `ITEM_BASED` expense already exists → returns a snapshot derived from the **expense**.<br />
• Otherwise → computes an *on-the-fly* summary using `Item.price` values (read-only).

Key Figures:
* `total_list_cost` – sum of item prices.
* `equal_share_per_user` – what each participant *should* pay.
* `balance` – (over / under) contribution for each user.

*Action:* `POST /costs/lists/{id}/cost-summary` finalises the list by persisting the derived `ITEM_BASED` expense.

### Group Balance Summary
Endpoint: `GET /costs/groups/{group_id}/balance-summary`

Aggregates across **all** group expenses + settlements:
* What each user paid (expenses + received settlements)
* What each user owed
* Suggested minimal settlement graph (creditors vs debtors)

---

## Data-Integrity Rules & Guards

1. `Expense` **must** reference *either* `list_id` **or** `group_id` (DB-level CHECK).
2. Row uniqueness guards: `UniqueConstraint('expense_id', 'user_id')` on `ExpenseSplit`.
3. `Settlement` payer ≠ payee (DB CHECK).
4. All mutating endpoints perform **authorization** checks via `crud_group` / `crud_list` helpers.
5. Monetary amounts are stored as `Numeric(10,2)` and rounded (`ROUND_HALF_UP`).

---

## Recent Fixes & Improvements (June 2025)

| Area | Issue | Resolution |
|------|-------|------------|
| **Recurring filter** | `GET /financials/expenses?isRecurring=true` referenced nonexistent `recurrence_rule`. | Switched to `is_recurring` flag. |
| **Enum mismatches** | `RecurrencePattern.type` stored uppercase enum, while API took lowercase strings. | Robust mapper converts strings → `RecurrenceTypeEnum`; scheduler is now case-insensitive. |
| **Scheduler** | `_calculate_next_occurrence` failed with Enum values & stringified `days_of_week`. | Added polymorphic handling + safe parsing of comma-separated strings. |

*All tests pass (`pytest -q`) and new unit tests cover the edge-cases above.*

---

## Extension Points

* **VAT / Tax logic** – attach a `tax_rate` column to `Expense` and resolve net vs gross amounts.
* **Multi-currency** – normalize amounts to a base currency using FX rates; expose a `Currency` table.
* **Budgeting / Limits** – per-group or per-list spending caps with alerting.
* **Webhook notifications** – publish `FinancialAuditLog` entries to external services.

---

## Appendix – Key SQL Schema Snapshots

```sql
-- Expense table (excerpt)
CREATE TABLE expenses (
    id              SERIAL PRIMARY KEY,
    total_amount    NUMERIC(10,2) NOT NULL,
    split_type      VARCHAR(20) NOT NULL,
    list_id         INT NULL,
    group_id        INT NULL,
    -- …
    CHECK (group_id IS NOT NULL OR list_id IS NOT NULL)
);

CREATE UNIQUE INDEX uq_expense_user_split ON expense_splits(expense_id, user_id);
```

---

## System Reliability Analysis & Improvements

### ✅ Implemented Reliability Features

#### 1. **Transaction Safety**
- All financial operations use **transactional sessions** (`get_transactional_session`)
- Atomic operations ensure data consistency across expense creation, splits, and settlements
- Row-level locking (`WITH FOR UPDATE`) prevents race conditions in settlement activities

#### 2. **Overpayment Protection** 
- **NEW**: Settlement activities now validate against remaining owed amount
- Prevents payments that exceed the split's owed amount
- Provides clear error messages with remaining balance information
- Handles multiple partial payments correctly

#### 3. **Data Validation & Constraints**
- **Decimal precision**: All monetary amounts use `Numeric(10,2)` with proper rounding
- **Positive amount validation**: Prevents negative payments and settlements
- **User existence validation**: Ensures all referenced users exist before operations
- **Split consistency**: Validates split totals match expense amounts (EXACT_AMOUNTS, PERCENTAGE)

#### 4. **Permission & Authorization**
- Multi-layered permission checks for expense creation and settlement recording
- Group owners can act on behalf of members with proper validation
- List/group access controls prevent unauthorized financial operations

#### 5. **Status Management**
- Automatic status updates for expense splits (unpaid → partially_paid → paid)
- Cascading status updates for parent expenses based on split states
- Pessimistic locking ensures consistent status transitions

#### 6. **Audit Trail**
- All financial operations logged via `create_financial_audit_log`
- Complete traceability of who created/modified financial records
- Immutable settlement activity records (no updates, only creation)

#### 7. **Error Handling**
- Comprehensive exception hierarchy for different error types
- Specific `OverpaymentError` for payment validation failures
- Database integrity and connection error handling
- Graceful degradation with meaningful error messages

### 🔍 Potential Areas for Enhancement

#### 1. **Optimistic Locking for Expenses**
Currently, expenses use basic versioning but could benefit from full optimistic locking:
```python
# Consider adding to ExpenseUpdate operations
if expected_version != current_expense.version:
    raise ConflictError("Expense was modified by another user")
```

#### 2. **Recurring Expense Reliability**
- Add retry logic for failed recurring expense generation
- Implement dead letter queue for failed recurring operations
- Add monitoring for recurring expense job health

#### 3. **Currency Consistency**
While currency is stored, there's no validation that all splits in an expense use the same currency:
```python
# Potential enhancement
if expense.currency != "USD" and any(split.currency != expense.currency for split in splits):
    raise InvalidOperationError("All splits must use the same currency as the parent expense")
```

#### 4. **Settlement Verification**
Consider adding verification flags for settlements to distinguish between:
- Automatic settlements (from expense splits)
- Manual settlements (direct user payments)
- Disputed settlements requiring verification

### 📊 System Health Metrics

The system provides comprehensive financial tracking through:

1. **Real-time balance calculations** via `costs_service.py`
2. **Settlement suggestions** using optimal debt reduction algorithms
3. **Expense categorization and filtering** with proper indexing
4. **Multi-context support** (lists, groups, personal expenses)

### 🛡️ Security Considerations

- **Input sanitization**: All financial inputs validated through Pydantic schemas
- **Authorization layers**: Multiple permission checks prevent unauthorized access
- **Audit logging**: Complete financial operation history for compliance
- **Data isolation**: Users only see expenses/settlements they have permission to access

### 🚀 Performance Optimizations

- **Database indexing** on critical foreign keys and search fields
- **Eager loading** with `selectinload` to prevent N+1 queries
- **Pagination** for large result sets
- **Connection pooling** with health checks (`pool_pre_ping=True`)

---

## Testing & Quality Assurance

The financial system includes comprehensive test coverage:

1. **Unit tests** for CRUD operations and business logic
2. **Integration tests** for API endpoints and workflows  
3. **Edge case testing** for overpayment protection and boundary conditions
4. **Concurrency tests** for settlement race conditions
5. **Data consistency tests** for split calculations and status updates

Example test scenarios:
- Multiple users settling the same split simultaneously
- Expense split totals validation across different split types
- Currency precision and rounding accuracy
- Permission boundary testing for cross-user operations

---

## Deployment & Monitoring Recommendations

### Database Considerations
- Regular backup strategy for financial data
- Monitor transaction isolation levels and deadlocks
- Set up alerts for unusual financial activity patterns
- Implement database connection monitoring

### Application Monitoring  
- Track settlement activity creation rates and failures
- Monitor recurring expense job execution and errors
- Set up alerts for permission denial patterns
- Track API response times for financial endpoints

### Business Intelligence
- Daily/weekly financial summaries per group
- Settlement velocity tracking (time to pay debts)
- Expense categorization analytics
- User engagement with financial features

The financial system is now **production-ready** with robust reliability safeguards, comprehensive error handling, and strong data consistency guarantees.