Skip to content

Market Index (pm-index)

Learning objectives

After reading this page you will understand:

  • What pm-index does and how it fits into the EduMatcher process model
  • How to start pm-index and which CLI options are available
  • How to configure one or more indices in the engine config (engine_config.yaml)
  • How cap-weighted index calculation and divisor normalisation work
  • How to apply corporate actions without disrupting the index level
  • How to query the current index value, structural/audit records, and level/EOD time-series history
  • Where state and history files are stored and how recovery works

What this process is

pm-index is a standalone index calculation process. It subscribes to trade events published by pm-engine and recomputes all configured indices in real time. Calculated values are re-published on a dedicated ZMQ socket so the market-data gateway (pm-md-gwy) can forward them to external subscribers over the CALF INDEX channel.

flowchart LR
    E["pm-engine\nPUB :5556"] -->|"trade.executed\nsession.state\nsystem.eod"| I["pm-index\nPUB :5558\nPULL :5559"]
    I -->|"index.update"| G["pm-md-gwy\nCALF TCP :5570"]
    G -->|"IDX messages"| C["External clients"]
    CON["pm-alf-console\n(INDEX|HISTORY)"] -->|"index.history_request"| I
    ADM["pm-index-admin-cli"] -->|"index.corp_action\nindex.constituent_change"| I

pm-index does not connect to the engine PULL socket. It never sends order or session commands to the engine — it only listens and publishes.

Prerequisites

  • pm-engine running
  • At least one index defined in engine_config.yaml, compiled and installed with pm-config-deploy (see Engine Configuration)
  • Every constituent symbol listed in symbols: with outstanding_shares set and at least one of last_buy_price / last_sell_price (its reference price)

pm-index can start before or after pm-engine; the ZMQ subscriber will reconnect automatically. Order only matters for the first few seconds of a fresh session — if pm-index misses early trades the reference prices from config are used as fallback until real trades arrive.

Starting pm-index

Installed mode:

pm-index
pm-index --reset

Developer / Poetry mode:

poetry run pm-index
poetry run pm-index --reset

pm-index takes no config-file argument. Like the engine, it reads the compiled configuration that pm-config-deploy installed (<DATA_DIR>/ref_data/engine_config.json), so it always agrees with the running engine about constituents and reference prices. If no compiled configuration is deployed it logs a warning (no compiled configuration at ... — no indices will be calculated. Run pm-config-deploy to install one.) and calculates nothing.

CLI options

Option Default Description
--reset off Delete persisted state files and re-initialise all indices from config
--version — Print the version and exit
--log-level WARNING Explicit level: CRITICAL, ERROR, WARNING, INFO, DEBUG
-v / --verbose off Increase verbosity (-v → INFO, -vv → DEBUG)
-q / --quiet off Accepted for symmetry with the other processes; WARNING is already the default
--log-target server Where this process's own log records go: server (auto-detected pm-log-srv), stdout, or file
--log-file — Operational log file path — required when --log-target file
--log-failover-timeout 30 Grace window in seconds before falling back to a local log file once pm-log-srv becomes unreachable

--reset is useful after:

  • changing constituent lists or base_value in config
  • replacing a corrupted state file
  • starting a new multi-day session from a clean baseline

Use --reset with care

--reset deletes the divisor and all persisted last prices for every configured index. The index restarts from scratch using only the reference prices in the engine config. The structural/audit JSONL file is not deleted; only the small state JSON file is removed.

Configuration

All index configuration lives in the top-level indices: section of engine_config.yaml (deployed with pm-config-deploy). See Engine Configuration for the full field reference; a complete YAML example is shown below.

Minimal example

symbols:
  AAPL:
    tick_decimals: 2
    outstanding_shares: 15000000000
  MSFT:
    tick_decimals: 2
    outstanding_shares: 7400000000
  TSLA:
    tick_decimals: 2
    outstanding_shares: 3200000000

participants:
  - id: TRADER01
    role: TRADER
    disconnect_behaviour: CANCEL_ALL

indices:
  - id: EDU100
    description: "EduMatcher broad index"
    base_value: 1000.0
    publish_interval_sec: 1.0
    history_file: data/indexes/EDU100_history.jsonl
    state_file: data/indexes/EDU100_state.json
    constituents:
      - AAPL
      - MSFT
      - TSLA

Field reference

Field Type Default Description
id string — Alphanumeric index identifier; used in events, file names, and gateway commands
description string — Human-readable label emitted in index events; required, non-empty
base_value float 1000.0 Starting index level on first launch; must be > 0
publish_interval_sec float 1.0 Throttle on how often index.update is broadcast; must be > 0
history_file string data/indexes/<ID>_history.jsonl Append-only JSONL structural/audit trail — corporate actions and constituent changes only, not level history (see State and History)
state_file string data/indexes/<ID>_state.json Checkpoint file for divisor and last prices
constituents list — Symbols included in the index

Constraints:

  • Maximum 5 indices per config file
  • id must be alphanumeric, and id values must be unique
  • constituents must be a non-empty list with no duplicates
  • Every constituent symbol must appear in symbols: with outstanding_shares set
  • Relative history_file / state_file paths (including a leading data/) are resolved inside the data directory (EDUMATCHER_DATA_DIR); absolute paths are used as given

Index eligibility is set at listing (IPO)

outstanding_shares is what makes a symbol index-eligible, and it is part of the symbol's initial listing (IPO). Set it when you list the symbol; a constituent without a positive outstanding_shares is rejected at startup. Share counts change only through corporate actions (splits, issuance), not intra-day trading.

Generating with pm-config-gen

pm-config-gen can emit the full indices: block automatically:

pm-config-gen \
  --symbols AAPL MSFT TSLA \
  --participants TRADER01 OPS01:ADMIN \
  --outstanding-shares AAPL:15000000000 \
  --outstanding-shares MSFT:7400000000 \
  --outstanding-shares TSLA:3200000000 \
  --sessions-enabled \
  --index EDU100:"EduMatcher broad index" \
  --index-constituents EDU100:AAPL,MSFT,TSLA \
  --output engine_config.yaml

File paths are derived from the index ID automatically when --index-history-file and --index-state-file are not specified.

The generated file has no reference prices yet

pm-index refuses a constituent that has neither last_buy_price nor last_sell_price, and pm-config-gen does not invent them. Either pass --seed-last-prices (emits null placeholders you then fill in by hand) or --seed-mm-mid-range MIN:MAX --seed-last-prices-from-mm (needs at least one MARKET_MAKER gateway), or add the two fields to each symbol yourself. Then install the result with pm-config-deploy engine_config.yaml.

How Calculation Works

Cap-weighted formula

Each index uses market-capitalisation weighting. A constituent's influence on the index level is proportional to its total market value relative to all other constituents:

\[ \text{aggregate\_cap} = \sum_{\text{sym} \in \text{constituents}} \text{last\_price}(\text{sym}) \times \text{outstanding\_shares}(\text{sym}) \]
\[ \text{index\_level} = \frac{\text{aggregate\_cap}}{\text{divisor}} \]

A company trading at $200 with 1 billion shares has a market cap of $200 billion. A company trading at $100 with 500 million shares has a market cap of $50 billion. The first company contributes four times as much to the index level as the second.

Divisor initialisation

On first startup (no persisted state), the divisor is chosen so the initial index level equals base_value:

\[ \text{initial\_divisor} = \frac{\text{aggregate\_cap at launch}}{\text{base\_value}} \]

Example with three stocks and base_value = 1000:

Symbol Price Outstanding shares Market cap
AAPL 209.50 15,000,000,000 3,142,500,000,000
MSFT 415.00 7,400,000,000 3,071,000,000,000
TSLA 248.00 3,200,000,000 793,600,000,000
Total 7,007,100,000,000 
initial_divisor = 7,007,100,000,000 / 1000.0 = 7,007,100,000

The divisor is a large number, but the ratio aggregate_cap / divisor always yields a level close to the configured base_value. After the first launch it evolves only through explicit corporate-action adjustments.

Update trigger

pm-index recalculates after every trade.executed message for a constituent symbol. Trades in non-constituent symbols are ignored. The calculation is O(N) in the number of constituents — fast enough for any educational exchange.

Reference prices

When no trade has occurred for a constituent since startup, a reference price seeded from the engine config is used: the average of last_buy_price and last_sell_price when both are set, or whichever one is set if only one is present. A symbol with neither field set fails to load as an index constituent. Once the first real trade arrives the reference price is replaced by the actual trade price.

Publish throttle

Recalculation happens on every trade, but publishing is throttled to publish_interval_sec (default 1 s). The internal level is always up to date; the throttle controls only how often index.update messages are broadcast.

Intraday OHLC

pm-index tracks day open, high, and low internally. They are cleared at every OPENING_AUCTION or CONTINUOUS session transition and start again from the next level update; the day is finalised with a closing value when the session reaches CLOSED.

Corporate Actions

Corporate actions change a company's share structure without destroying value. Without divisor adjustment, a 2-for-1 stock split would make the index drop by roughly half — which would be wrong because no value was lost.

The divisor is adjusted so the index level is preserved at the moment of the action. Future price movements then reflect real value changes.

Divisor adjustment formula

\[ \text{new\_divisor} = \text{old\_divisor} \times \frac{\text{new\_aggregate\_cap}}{\text{old\_aggregate\_cap}} \]

This guarantees continuity:

\[ \frac{\text{new\_cap}}{\text{new\_divisor}} = \frac{\text{new\_cap}}{\text{old\_divisor} \times \frac{\text{new\_cap}}{\text{old\_cap}}} = \frac{\text{old\_cap}}{\text{old\_divisor}} \quad \checkmark \]

Supported corporate actions

Action action value Required parameters Effect
Stock split SPLIT ratio_numerator, ratio_denominator Increases shares, reduces price by the same factor; divisor corrected for integer-rounding drift
Cash dividend CASH_DIVIDEND dividend_per_share Reduces price by the dividend amount; divisor adjusted to preserve index level
Shares issuance SHARES_ISSUANCE new_shares_outstanding Sets the new total share count outright (not a delta); divisor rescaled to preserve index level — typically upward for a genuine issuance, but the code does not require new_shares_outstanding to exceed the current count, so supplying a smaller value rescales the divisor downward instead

Applying corporate actions

Use pm-index-admin-cli to apply corporate actions to a running pm-index process:

pm-index-admin-cli --id OPS01 split    --index EDU100 --sym AAPL --ratio 2:1
pm-index-admin-cli --id OPS01 dividend --index EDU100 --sym MSFT --amount 2.50
pm-index-admin-cli --id OPS01 shares   --index EDU100 --sym TSLA --new-shares 3500000000
pm-index-admin-cli --id OPS01 shares   --index EDU100 --sym TSLA --delta -200000000   # buy-back

Each mutating subcommand prints a confirmation prompt before sending (skip it with -y/--yes), supports --dry-run to preview the outbound payload without sending it, and blocks for the index.corp_action_ack.{gateway_id} response. pm-index applies the action in-process, immediately publishes an updated index value live, and writes a CORP_ACTION record to the structural/audit history file. See Index Admin CLI for the full subcommand reference, --dry-run output, and exit codes.

Apply corporate actions before the market opens

It is safest to apply corporate actions during PRE_OPEN before trading starts for the day. Applying them during continuous trading means the index will have a brief moment where the pre-adjustment divisor is used for one more trade before the action takes effect.

Under the hood: ExchangeCommandClient

pm-index-admin-cli is a thin wrapper over the ExchangeCommandClient class (edumatcher.commands.client) — the same class pm-admin-cli uses internally for its engine commands. The CLI calls its index_corp_action() method directly:

from edumatcher.commands import ExchangeCommandClient

client = ExchangeCommandClient("GW_ADMIN")

client.index_corp_action(
    "EDU100", "SPLIT", "AAPL",
    ratio_numerator=2, ratio_denominator=1,
)
client.index_corp_action(
    "EDU100", "CASH_DIVIDEND", "MSFT",
    dividend_per_share=2.50,
)
client.index_corp_action(
    "EDU100", "SHARES_ISSUANCE", "TSLA",
    new_shares_outstanding=3500000000,
)

client.close()

Unlike the engine's PUSH socket, pm-index's PULL socket has no connect()/auth handshake — see Index Admin CLI — Why there is no connect() step. Writing a script against this class directly still works and is how pm-index-admin-cli itself is implemented, but the CLI adds validation, a confirmation prompt, and --dry-run, so it's the preferred path for interactive/manual use.

Constituent changes

Constituents can be added or removed without restarting pm-index, also via pm-index-admin-cli:

pm-index-admin-cli --id OPS01 add    --index EDU100 --sym AMZN --shares 10500000000 --price 195.00
pm-index-admin-cli --id OPS01 delist --index EDU100 --sym TSLA

Adding a constituent adjusts the divisor so the index level does not change at the moment of addition. Delisting a constituent adjusts the divisor so the remaining constituents continue smoothly.

Every corporate action and constituent change is written to the history_file as a CORP_ACTION, ADD_CONSTITUENT, or DELIST record — queryable with pm-index-cli.

State and History

State file

pm-index writes a small JSON checkpoint file at startup (right after computing the initial level), after every EOD finalization, and after any corporate action or constituent change (add/delist):

{
  "index_id": "EDU100",
  "description": "EduMatcher broad index",
  "divisor": 7007100000.0,
  "constituents": ["AAPL", "MSFT", "TSLA"],
  "last_prices": {
    "AAPL": 211.3,
    "MSFT": 418.5,
    "TSLA": 251.0
  },
  "day_open": 1002.45,
  "day_high": 1012.35,
  "day_low": 998.4,
  "last_level": 1008.9195,
  "last_updated": 1749760800.0
}

On restart, this file is loaded to restore the divisor and last prices. If the index_id in the file does not match, or the persisted constituent list differs from the configured one, pm-index exits with an error telling you to use --reset. The persisted list is stored sorted alphabetically and is compared in order with the configured constituents, so list constituents alphabetically in the config. After an add or delist made through pm-index-admin-cli, update the config to match, deploy it, and start with --reset (the change is otherwise lost or trips this check on the next start).

History file

The JSONL history file is a structural/corporate-action audit log, not a level-history store. Only events that change an index's composition or divisor are appended (one JSON object per line):

Record type When written
INIT First-ever startup (no prior state); records base_value, divisor, constituent list
CORP_ACTION After every split, dividend, or shares-issuance adjustment
ADD_CONSTITUENT After a new symbol is added to the index
DELIST After a symbol is removed from the index
REBALANCE After a batch share-outstanding update

Every record carries type, ts_ns (epoch nanoseconds), index_id and level. Records are written as compact JSON (no spaces after : or ,). Example history file extract:

{"type":"INIT","ts_ns":1749733100000000000,"index_id":"EDU100","base_value":1000.0,"divisor":7007100000.0,"constituents":["AAPL","MSFT","TSLA"],"level":1000.0}
{"type":"CORP_ACTION","ts_ns":1749847200000000000,"index_id":"EDU100","symbol":"AAPL","action":"SPLIT","detail":"2:1","old_divisor":7007100000.0,"new_divisor":7007100000.0,"level":1051.2}

History files are not deleted by --reset. They accumulate across sessions and provide a permanent structural audit trail.

Level and EOD history live in pm-stats, not here

Every index.update broadcast — including the throttled intraday ticks and the forced end-of-day publish — is recorded by pm-stats in two SQLite tables: index_level_snapshots (every tick) and index_daily_stats (daily OHLC rollup). Query them with pm-stats-cli index-snapshots and pm-stats-cli index-daily. This JSONL file intentionally does not duplicate that data; it exists purely so operators can audit why an index's divisor changed (splits, dividends, constituent changes) without scanning a high-frequency time series.

Note that index_daily_stats.close_level reflects the most recent update for that date, not necessarily the final close, until the day ends or close_session_state reads CLOSED — see Getting the EOD index level for a date.

Getting Index Values

From a gateway terminal (INDEX command)

Connect through any pm-alf-console session (including ADMIN role) and run:

INDEX

Sample output:

[10:15:23.411] EDU100 1048.73 +6.63 +0.64% O=1042.10 H=1056.30 L=1040.05 CONTINUOUS

pm-index must be running. If it is not, the gateway prints "No index data received yet. Is pm-index running?"

Structural/audit history queries

INDEX|HISTORY|INDEX=EDU100

INDEX=<id> is required unless the console has already received at least one index.update broadcast for an index in this session, in which case that index's ID is used as the default and INDEX= may be omitted. Without either, the console prints INDEX|HISTORY requires INDEX=<id> or prior index.update. and sends nothing.

Returns the last 30 days of structural/audit records (INIT, CORP_ACTION, ADD_CONSTITUENT, DELIST, REBALANCE) — corporate actions and constituent changes, not level ticks. The result is printed as a table titled Index structural history with the columns Type, Time (local), Symbol, Detail and Level; when nothing matches it prints a "No structural index history records returned" hint instead.

INDEX|HISTORY|INDEX=EDU100|FROM=2026-06-01|TO=2026-06-12

Returns records within the given date range, oldest first. FROM and TO are dates (YYYY-MM-DD) read as local midnight at the start of that day, so TO=2026-06-12 stops at the start of 12 June and does not include that day's records — use the following day to include it. An unparseable date is silently ignored and the default (30 days back / now) is used.

Level/EOD history is a different tool

INDEX|HISTORY only returns structural/audit events. For index level or end-of-day time-series data, use pm-stats-cli index-snapshots or pm-stats-cli index-daily instead (see Statistics and Reporting).

Offline analysis with pm-index-cli

For richer filtering, CSV/JSON export, or scripting of structural/audit records, use pm-index-cli instead. It reads the JSONL history files directly without going through the gateway socket.

From the market-data gateway (CALF)

If pm-md-gwy is running and connected to pm-index, external clients can subscribe via the CALF INDEX channel:

HELLO|CLIENT=student01|PROTO=CALF1
SUB|CH=INDEX|SYM=EDU100

The gateway sends an initial SNAP and then live IDX messages:

SNAP|CH=INDEX|SYM=EDU100|SEQ=1|TS=2026-06-12T10:15:23.000Z|LEVEL=1048.73|SESSION=CONTINUOUS|OPEN=1042.1|CHG=+6.63|PCTCHG=+0.64|HIGH=1056.3|LOW=1040.05|AGGCAP=7348555983000
IDX|CH=INDEX|SYM=EDU100|SEQ=2|TS=2026-06-12T10:15:24.411Z|LEVEL=1051.2|SESSION=CONTINUOUS|OPEN=1042.1|CHG=+9.10|PCTCHG=+0.87|HIGH=1056.3|LOW=1040.05|AGGCAP=7365863520000

The SNAP carries the same fields as the most recent IDX. LEVEL, OPEN, HIGH and LOW are the raw numbers (no fixed decimal places — 1042.1, not 1042.10); CHG and PCTCHG are rounded to two places. The field order on the wire is as shown, but clients should parse by key.

IDX message fields

Field Type Description
CH string Always INDEX
SYM string Index ID (e.g. EDU100)
SEQ int Monotonic sequence number for gap detection
TS string UTC ISO-8601 timestamp
LEVEL decimal Current index level
CHG decimal Change from day open (signed); omitted before first open
PCTCHG decimal Percentage change from day open (signed); omitted before first open
OPEN decimal Day open level; omitted until the first level update after the day's open/high/low were last cleared (each OPENING_AUCTION / CONTINUOUS transition)
HIGH decimal Day high; omitted together with OPEN
LOW decimal Day low; omitted together with OPEN
AGGCAP int Current aggregate market cap
SESSION string Current session state

Using pm-index-cli for structural/audit records

pm-index-cli queries the structural/audit JSONL files directly from the command line — no running pm-index process is needed. It does not have level or EOD subcommands; use pm-stats-cli for that (next section).

--config, --data-dir, --format, and --no-header are global options and must come before the subcommand (events / indices); putting them after the subcommand is a parse error since only the top-level parser defines them.

# Corporate actions and constituent changes (all time)
pm-index-cli --config engine_config.yaml events --index EDU100

# Filter to a specific structural event type
pm-index-cli --config engine_config.yaml events --index EDU100 --type CORP_ACTION

# Export to CSV for import into a spreadsheet (--format comes before the subcommand)
pm-index-cli --config engine_config.yaml --format csv events > edu100_events.csv

# JSON output for scripting
pm-index-cli --config engine_config.yaml --format json events --index EDU100 \
  | python3 -c "import json,sys; [print(r['ts'], r['type'], r['detail']) for r in json.load(sys.stdin)]"

# Date-range query
pm-index-cli --config engine_config.yaml events \
  --index EDU100 --from 2026-06-01 --to 2026-06-25

# List all configured indices
pm-index-cli --config engine_config.yaml indices

# --config is optional: without it the deployed (compiled) configuration is used.
# --data-dir is only a fallback for indices the configuration does not know
# (default data/indexes, inside the data directory); a configured history_file wins.
pm-index-cli --data-dir /custom/path/indexes events --index EDU100

See the pm-index-cli reference for all subcommands, column descriptions, and output-format options.

Querying level/EOD history with pm-stats-cli

For index level ticks or daily OHLC history, use pm-stats-cli instead, which reads from pm-stats' SQLite database:

# Daily open/high/low/close rollup
pm-stats-cli index-daily --index-id EDU100

# Raw level snapshots (every recorded tick) in a time window
pm-stats-cli index-snapshots --index-id EDU100 --from 2026-06-01 --to 2026-06-25

# List all index IDs that have recorded data
pm-stats-cli index-ids

See Statistics and Reporting for the full command reference.

Via the REST API

pm-api-gwy also exposes both the structural/audit log and the level/EOD time series over HTTP, under /api/v1/history:

Endpoint Data Backing store
GET /history/index-daily Daily index OHLC rows pm-stats SQLite
GET /history/index-snapshots Intraday index level ticks pm-stats SQLite
GET /history/index-ids Index IDs with recorded statistics pm-stats SQLite
GET /history/index-events Structural/audit records (INIT, CORP_ACTION, ADD_CONSTITUENT, DELIST, REBALANCE) Live round-trip to pm-index, not pm-stats 
curl -s -H "Authorization: Bearer key-trader-demo" \
  "http://127.0.0.1:8080/api/v1/history/index-daily?index_id=EDU100&date=2026-06-14"

curl -s -H "Authorization: Bearer key-trader-demo" \
  "http://127.0.0.1:8080/api/v1/history/index-events?index_id=EDU100&from=1750000000&to=1760000000"

See API Gateway — History endpoints for the full parameter reference, pagination rules, and the note that index-events is the one history endpoint that is not served from pm-stats and has no pagination cursor.

Reading the structural/audit file directly

The JSONL file is plain text — one JSON object per line. This can be useful for quick one-off inspections or when pm-index-cli is not available:

tail -20 data/indexes/EDU100_history.jsonl

# Corporate action records only
grep '"type":"CORP_ACTION"' data/indexes/EDU100_history.jsonl | python3 -m json.tool

# Structural records between two timestamps
python3 - <<'EOF'
import json
from_ts = 1749700000.0
to_ts   = 1749800000.0
with open("data/indexes/EDU100_history.jsonl") as f:
    for line in f:
        rec = json.loads(line)
        ts = rec["ts_ns"] / 1e9          # epoch nanoseconds -> seconds
        if from_ts <= ts <= to_ts:
            print(f"{ts:.0f}  {rec['type']}  {rec.get('symbol', '')}")
EOF

Example Configurations

The pm-config-gen commands below need the reference prices described in the warning above (for example --seed-last-prices) before pm-index will accept the result.

Single index, three constituents

pm-config-gen \
  --symbols AAPL MSFT TSLA \
  --participants TRADER01 TRADER02 OPS01:ADMIN \
  --outstanding-shares AAPL:15000000000 \
  --outstanding-shares MSFT:7400000000 \
  --outstanding-shares TSLA:3200000000 \
  --sessions-enabled \
  --index EDU100:"EduMatcher broad index" \
  --index-constituents EDU100:AAPL,MSFT,TSLA \
  --output engine_config.yaml

Two indices with different base values and publish rates

pm-config-gen \
  --symbols AAPL MSFT TSLA \
  --participants TRADER01 OPS01:ADMIN \
  --outstanding-shares AAPL:15000000000 \
  --outstanding-shares MSFT:7400000000 \
  --outstanding-shares TSLA:3200000000 \
  --index BROAD:"Broad market" \
  --index-constituents BROAD:AAPL,MSFT,TSLA \
  --index-base-value BROAD:1000.0 \
  --index TECH2:"Technology pair" \
  --index-constituents TECH2:AAPL,MSFT \
  --index-base-value TECH2:500.0 \
  --index-interval TECH2:2.0 \
  --output engine_config.yaml

Full manual YAML for two indices

indices:
  - id: BROAD
    description: "Broad market"
    base_value: 1000.0
    publish_interval_sec: 1.0
    history_file: data/indexes/BROAD_history.jsonl
    state_file: data/indexes/BROAD_state.json
    constituents:
      - AAPL
      - MSFT
      - TSLA

  - id: TECH2
    description: "Technology pair"
    base_value: 500.0
    publish_interval_sec: 2.0
    history_file: data/indexes/TECH2_history.jsonl
    state_file: data/indexes/TECH2_state.json
    constituents:
      - AAPL
      - MSFT

Operational Checklist

  1. Ensure every constituent has outstanding_shares set in symbols:.
  2. Create the data/indexes/ directory before first run (or let pm-index create it automatically — it creates parent directories as needed).
  3. Start pm-engine before pm-index during normal operations; pm-index will reconnect if the order is reversed but may miss the first few trades.
  4. Use --reset only when deliberately changing the constituent list or base_value. This discards the divisor history for all indices.
  5. Apply corporate actions during PRE_OPEN to avoid mid-session index jumps, using pm-index-admin-cli.
  6. Check data/indexes/<ID>_state.json after a crash to confirm the divisor was persisted. If the file is missing or truncated, use --reset.

Keeping history - Architecture Overview

Historic values of a market index gets a bit complex as along with the index vaues there is also the need to track included symbols as well as corporate actions.

There are three separate persistence layers for market index data:

1. Structural Events (JSONL Append-Only Log)

  • File: history_file (configured per index)
  • Manager: IndexHistory class in history.py
  • Content: Structural/corporate-action events only
  • INIT: Index initialization
  • CORP_ACTION: Splits, dividends, share issuance
  • DELIST: Constituent removal
  • ADD_CONSTITUENT: New constituent addition
  • REBALANCE: Batch share-outstanding updates
  • Format: JSON Lines (one record per line)
  • Access: Read-only scan from oldest record to requested time range

2. Per-Tick Level History (SQLite)

  • Tables: index_level_snapshots and index_daily_stats
  • Manager: pm-stats process (NOT pm-index)
  • Content: Per-tick level updates, OHLC (Open/High/Low/Close) data
  • Note: This is the real home for level-update history

3. Current Index State (JSON File)

  • File: state_file (configured per index)
  • Content: Latest state snapshot
  • Divisor value
  • Last prices for each constituent
  • Day OHLC summary
  • Last calculated level
  • Last update timestamp
  • Purpose: Restart recovery (load-on-startup)