0207c175ba
This commit introduces a new endpoint for updating chores of any type, allowing conversions between personal and group chores while enforcing permission checks. Additionally, structured logging has been implemented through a new middleware, improving request tracing and logging details for better monitoring and debugging. These changes aim to enhance the functionality and maintainability of the chore management system.
270 lines
12 KiB
Markdown
270 lines
12 KiB
Markdown
# Financial System Overview
|
||
|
||
## Introduction
|
||
|
||
This document provides a comprehensive overview of the **Expense, Cost & Financial** domain of the project. It is intended for backend / frontend developers, QA engineers and DevOps personnel who need to understand **how money flows through the system**, what invariants are enforced and where to extend the platform.
|
||
|
||
> **TL;DR** The financial subsystem is a *Split-wise*-inspired engine with first-class support for: shared lists, item-derived expenses, multi-scheme splitting, settlements, recurring charges and complete auditability.
|
||
|
||
---
|
||
|
||
## Main Concepts & Entities
|
||
|
||
| Entity | Purpose | Key Relationships |
|
||
|--------|---------|-------------------|
|
||
| **User** | A registered account. Owns expenses, owes splits, records settlements. | `expenses_paid`, `expenses_created`, `expense_splits`, `settlements_made/received`, `settlements_created` |
|
||
| **Group** | A collection of users with shared expenses / lists. | `member_associations (UserGroup)`, `lists`, `expenses`, `settlements` |
|
||
| **List** | A shopping / to-do list. May belong to a `Group` or be personal. | `items`, `expenses` |
|
||
| **Item** | A purchasable line inside a `List`. Price & author drive *item-based* expense splits. | `list`, `added_by_user` |
|
||
| **Expense** | A monetary outflow. | `splits`, `list`, `group`, `item`, `recurrence_pattern` |
|
||
| **ExpenseSplit** | A *who-owes-what* record for an `Expense`. | `user`, `settlement_activities` |
|
||
| **Settlement** | A generic *cash transfer* between two users inside a group. | – |
|
||
| **SettlementActivity** | A *payment* that reduces an individual `ExpenseSplit` (e.g. Alice pays Bob her part). | – |
|
||
| **RecurrencePattern** | The schedule template that spawns future `Expense` occurrences. | `expenses` |
|
||
| **FinancialAuditLog** | Append-only journal of *who did what* (create / update / delete) for all financial entities. | – |
|
||
|
||
---
|
||
|
||
## Expense Lifecyle
|
||
|
||
1. **Creation** (`POST /financials/expenses`)
|
||
• Caller provides an `ExpenseCreate` DTO.<br />
|
||
• Backend validates the *context* (list / group / item), the *payer*, and the chosen **split strategy**.
|
||
• Supported `split_type` values (`SplitTypeEnum`):
|
||
* `EQUAL` – evenly divided among computed participants.
|
||
* `EXACT_AMOUNTS` – caller supplies absolute owed amounts.
|
||
* `PERCENTAGE` – caller supplies percentages totalling 100 %.
|
||
* `SHARES` – integer share units (e.g. 1 : 2 : 3).
|
||
* `ITEM_BASED` – derived from priced `Item`s in a list.
|
||
• A database transaction writes `Expense` + `ExpenseSplit` rows and a `FinancialAuditLog` entry.
|
||
|
||
2. **Reading**
|
||
• `GET /financials/expenses/{id}` enforces *row-level* security: the requester must be payer, list member or group member.
|
||
|
||
3. **Update / Delete**
|
||
• Optimistic-locking via the `version` field.
|
||
• Only the **payer** or a **group owner** may mutate records.
|
||
|
||
4. **Settlement**
|
||
• Generic settlements (`/financials/settlements`) clear balances between two users.<br />
|
||
• Fine-grained settlements (`/financials/expense_splits/{id}/settle`) clear a single `ExpenseSplit`.
|
||
|
||
5. **Recurring Expenses**
|
||
• An `Expense` can be flagged `is_recurring = true` and carry a `RecurrencePattern`.<br />
|
||
• A background job (`app/jobs/recurring_expenses.py::generate_recurring_expenses`) wakes up daily and:
|
||
1. Finds template expenses due (`next_occurrence <= now`).
|
||
2. Spawns a *child* expense replicating the split logic.
|
||
3. Updates `last_occurrence`, decrements `max_occurrences` and calculates the next date.
|
||
|
||
---
|
||
|
||
## Cost Summaries & Balance Sheets
|
||
|
||
### List Cost Summary
|
||
Endpoint: `GET /costs/lists/{list_id}/cost-summary`
|
||
|
||
• If an `ITEM_BASED` expense already exists → returns a snapshot derived from the **expense**.<br />
|
||
• Otherwise → computes an *on-the-fly* summary using `Item.price` values (read-only).
|
||
|
||
Key Figures:
|
||
* `total_list_cost` – sum of item prices.
|
||
* `equal_share_per_user` – what each participant *should* pay.
|
||
* `balance` – (over / under) contribution for each user.
|
||
|
||
*Action:* `POST /costs/lists/{id}/cost-summary` finalises the list by persisting the derived `ITEM_BASED` expense.
|
||
|
||
### Group Balance Summary
|
||
Endpoint: `GET /costs/groups/{group_id}/balance-summary`
|
||
|
||
Aggregates across **all** group expenses + settlements:
|
||
* What each user paid (expenses + received settlements)
|
||
* What each user owed
|
||
* Suggested minimal settlement graph (creditors vs debtors)
|
||
|
||
---
|
||
|
||
## Data-Integrity Rules & Guards
|
||
|
||
1. `Expense` **must** reference *either* `list_id` **or** `group_id` (DB-level CHECK).
|
||
2. Row uniqueness guards: `UniqueConstraint('expense_id', 'user_id')` on `ExpenseSplit`.
|
||
3. `Settlement` payer ≠ payee (DB CHECK).
|
||
4. All mutating endpoints perform **authorization** checks via `crud_group` / `crud_list` helpers.
|
||
5. Monetary amounts are stored as `Numeric(10,2)` and rounded (`ROUND_HALF_UP`).
|
||
|
||
---
|
||
|
||
## Recent Fixes & Improvements (June 2025)
|
||
|
||
| Area | Issue | Resolution |
|
||
|------|-------|------------|
|
||
| **Recurring filter** | `GET /financials/expenses?isRecurring=true` referenced nonexistent `recurrence_rule`. | Switched to `is_recurring` flag. |
|
||
| **Enum mismatches** | `RecurrencePattern.type` stored uppercase enum, while API took lowercase strings. | Robust mapper converts strings → `RecurrenceTypeEnum`; scheduler is now case-insensitive. |
|
||
| **Scheduler** | `_calculate_next_occurrence` failed with Enum values & stringified `days_of_week`. | Added polymorphic handling + safe parsing of comma-separated strings. |
|
||
|
||
*All tests pass (`pytest -q`) and new unit tests cover the edge-cases above.*
|
||
|
||
---
|
||
|
||
## Extension Points
|
||
|
||
* **VAT / Tax logic** – attach a `tax_rate` column to `Expense` and resolve net vs gross amounts.
|
||
* **Multi-currency** – normalize amounts to a base currency using FX rates; expose a `Currency` table.
|
||
* **Budgeting / Limits** – per-group or per-list spending caps with alerting.
|
||
* **Webhook notifications** – publish `FinancialAuditLog` entries to external services.
|
||
|
||
---
|
||
|
||
## Appendix – Key SQL Schema Snapshots
|
||
|
||
```sql
|
||
-- Expense table (excerpt)
|
||
CREATE TABLE expenses (
|
||
id SERIAL PRIMARY KEY,
|
||
total_amount NUMERIC(10,2) NOT NULL,
|
||
split_type VARCHAR(20) NOT NULL,
|
||
list_id INT NULL,
|
||
group_id INT NULL,
|
||
-- …
|
||
CHECK (group_id IS NOT NULL OR list_id IS NOT NULL)
|
||
);
|
||
|
||
CREATE UNIQUE INDEX uq_expense_user_split ON expense_splits(expense_id, user_id);
|
||
```
|
||
|
||
---
|
||
|
||
## System Reliability Analysis & Improvements
|
||
|
||
### ✅ Implemented Reliability Features
|
||
|
||
#### 1. **Transaction Safety**
|
||
- All financial operations use **transactional sessions** (`get_transactional_session`)
|
||
- Atomic operations ensure data consistency across expense creation, splits, and settlements
|
||
- Row-level locking (`WITH FOR UPDATE`) prevents race conditions in settlement activities
|
||
|
||
#### 2. **Overpayment Protection**
|
||
- **NEW**: Settlement activities now validate against remaining owed amount
|
||
- Prevents payments that exceed the split's owed amount
|
||
- Provides clear error messages with remaining balance information
|
||
- Handles multiple partial payments correctly
|
||
|
||
#### 3. **Data Validation & Constraints**
|
||
- **Decimal precision**: All monetary amounts use `Numeric(10,2)` with proper rounding
|
||
- **Positive amount validation**: Prevents negative payments and settlements
|
||
- **User existence validation**: Ensures all referenced users exist before operations
|
||
- **Split consistency**: Validates split totals match expense amounts (EXACT_AMOUNTS, PERCENTAGE)
|
||
|
||
#### 4. **Permission & Authorization**
|
||
- Multi-layered permission checks for expense creation and settlement recording
|
||
- Group owners can act on behalf of members with proper validation
|
||
- List/group access controls prevent unauthorized financial operations
|
||
|
||
#### 5. **Status Management**
|
||
- Automatic status updates for expense splits (unpaid → partially_paid → paid)
|
||
- Cascading status updates for parent expenses based on split states
|
||
- Pessimistic locking ensures consistent status transitions
|
||
|
||
#### 6. **Audit Trail**
|
||
- All financial operations logged via `create_financial_audit_log`
|
||
- Complete traceability of who created/modified financial records
|
||
- Immutable settlement activity records (no updates, only creation)
|
||
|
||
#### 7. **Error Handling**
|
||
- Comprehensive exception hierarchy for different error types
|
||
- Specific `OverpaymentError` for payment validation failures
|
||
- Database integrity and connection error handling
|
||
- Graceful degradation with meaningful error messages
|
||
|
||
### 🔍 Potential Areas for Enhancement
|
||
|
||
#### 1. **Optimistic Locking for Expenses**
|
||
Currently, expenses use basic versioning but could benefit from full optimistic locking:
|
||
```python
|
||
# Consider adding to ExpenseUpdate operations
|
||
if expected_version != current_expense.version:
|
||
raise ConflictError("Expense was modified by another user")
|
||
```
|
||
|
||
#### 2. **Recurring Expense Reliability**
|
||
- Add retry logic for failed recurring expense generation
|
||
- Implement dead letter queue for failed recurring operations
|
||
- Add monitoring for recurring expense job health
|
||
|
||
#### 3. **Currency Consistency**
|
||
While currency is stored, there's no validation that all splits in an expense use the same currency:
|
||
```python
|
||
# Potential enhancement
|
||
if expense.currency != "USD" and any(split.currency != expense.currency for split in splits):
|
||
raise InvalidOperationError("All splits must use the same currency as the parent expense")
|
||
```
|
||
|
||
#### 4. **Settlement Verification**
|
||
Consider adding verification flags for settlements to distinguish between:
|
||
- Automatic settlements (from expense splits)
|
||
- Manual settlements (direct user payments)
|
||
- Disputed settlements requiring verification
|
||
|
||
### 📊 System Health Metrics
|
||
|
||
The system provides comprehensive financial tracking through:
|
||
|
||
1. **Real-time balance calculations** via `costs_service.py`
|
||
2. **Settlement suggestions** using optimal debt reduction algorithms
|
||
3. **Expense categorization and filtering** with proper indexing
|
||
4. **Multi-context support** (lists, groups, personal expenses)
|
||
|
||
### 🛡️ Security Considerations
|
||
|
||
- **Input sanitization**: All financial inputs validated through Pydantic schemas
|
||
- **Authorization layers**: Multiple permission checks prevent unauthorized access
|
||
- **Audit logging**: Complete financial operation history for compliance
|
||
- **Data isolation**: Users only see expenses/settlements they have permission to access
|
||
|
||
### 🚀 Performance Optimizations
|
||
|
||
- **Database indexing** on critical foreign keys and search fields
|
||
- **Eager loading** with `selectinload` to prevent N+1 queries
|
||
- **Pagination** for large result sets
|
||
- **Connection pooling** with health checks (`pool_pre_ping=True`)
|
||
|
||
---
|
||
|
||
## Testing & Quality Assurance
|
||
|
||
The financial system includes comprehensive test coverage:
|
||
|
||
1. **Unit tests** for CRUD operations and business logic
|
||
2. **Integration tests** for API endpoints and workflows
|
||
3. **Edge case testing** for overpayment protection and boundary conditions
|
||
4. **Concurrency tests** for settlement race conditions
|
||
5. **Data consistency tests** for split calculations and status updates
|
||
|
||
Example test scenarios:
|
||
- Multiple users settling the same split simultaneously
|
||
- Expense split totals validation across different split types
|
||
- Currency precision and rounding accuracy
|
||
- Permission boundary testing for cross-user operations
|
||
|
||
---
|
||
|
||
## Deployment & Monitoring Recommendations
|
||
|
||
### Database Considerations
|
||
- Regular backup strategy for financial data
|
||
- Monitor transaction isolation levels and deadlocks
|
||
- Set up alerts for unusual financial activity patterns
|
||
- Implement database connection monitoring
|
||
|
||
### Application Monitoring
|
||
- Track settlement activity creation rates and failures
|
||
- Monitor recurring expense job execution and errors
|
||
- Set up alerts for permission denial patterns
|
||
- Track API response times for financial endpoints
|
||
|
||
### Business Intelligence
|
||
- Daily/weekly financial summaries per group
|
||
- Settlement velocity tracking (time to pay debts)
|
||
- Expense categorization analytics
|
||
- User engagement with financial features
|
||
|
||
The financial system is now **production-ready** with robust reliability safeguards, comprehensive error handling, and strong data consistency guarantees. |