Cellpy Core Harmonized_Raw¶

Authoritative definition — this is the single source of truth for the harmonized raw format. (Moved here from docs/harmonized_raw_definition.md; it supersedes the older 2025-09-08 draft that was removed in issue #10.)

Date: 2025-09-17 (reviewed in issue #10)
SPEED-16: "Define the object that goes into cellpy core"

Purpose¶

Original cycler files are converted to this format, in order to unify the format so that only one importer has to be used wherever it will be imported.

Conventions¶

• The cell voltage column is named potential (not voltage).
• Time columns (test_time, etc.) are in seconds.
• The cycler step mode column is step_mode. There is no channel_status column (it was dropped in issue #10).
• epoch_time_utc is int64 nanoseconds since the Unix epoch, UTC. This matches the internal representation of polars/pandas/Arrow timestamps, so round-trips to native datetime types are lossless (float epoch seconds cannot hold nanosecond resolution). When an importer only has the cycler's wall-clock timestamp (often a naive, timezone-less datetime, e.g. legacy Arbin date_time), it is converted to epoch nanoseconds by treating the naive timestamp as UTC. (Example: dev/make_harmonized_raw.py does this when building the cycler_cc_harmonized_raw.parquet test fixture.) Conversion helpers to/from float seconds UTC and native datetime live in cellpycore.timestamps.
• Capacity / energy columns are cumulative per cycle, per direction — see Capacity convention below. (Renamed from step_cumulative_* in issue #13; the old name was misleading.)

Column Headers¶

Set up a flexible structure that allows for more columns.

** calculate if empty; keep if filled (can be overridden by argument)

Capacity convention¶

cumulative_charge_capacity / cumulative_discharge_capacity (and the matching energy columns) are cumulative per cycle, per direction: within a cycle they accumulate across that cycle's steps for their direction, and reset to 0 at each cycle boundary. This matches how legacy cellpy raw data is supplied (verified on real Arbin data in issue #13) and is what the summary path relies on (the per-cycle capacity is read from the cycle's last datapoint).

Downstream values are derived, not stored in raw: - per-step capacity = the in-step delta (last − first within a step group), produced in the step table; - per-cycle capacity = the cycle-end cumulative value, produced in the cycle/summary table.

Importers must deliver cycle-cumulative capacity. (Accepting step-/test-cumulative raw and normalizing the reset granularity inside the engine is a possible future extension.)

Auxillary columns¶

Option for more auxillary columns; naming scheme: - aux_temperature_[arbitrary name] - aux_potential_[] - aux_pressure_[] - aux_resistance_[]

Test metadata (TestMeta)¶

Per-test metadata, one record per test, keyed by test_id (the same test_id carried on every raw row). This is where test-level descriptors live so they are not repeated on every raw row; it is what lets a single object hold many merged test files efficiently (see .issueflows/04-designs-and-guides/test-metadata-and-merging.md). When several tests are merged, TestMeta holds one row per test_id.

Test-level descriptors that are constant within a test (e.g. test_family, test_type, cycle_mode) belong here, not in the raw columns.

This spec remains authoritative for the field set. The fields below now also have a code home as the TestMeta / CellMeta dataclasses in cellpycore.metadata (scaffolding; see .issueflows/04-designs-and-guides/metadata-scaffolding.md, issue #30).

Cell / material metadata (optional). Physical cell properties (mass, total mass, nominal capacity + specifics, active-electrode area / loading / thickness, electrode and electrolyte types, experiment type, ...) are also per-test/per-cell. Legacy cellpy already defines these in cellpy.parameters.internal_settings.CellpyMetaCommon (cell/material/ geometry) and CellpyMetaIndividualTest (test-dependent: channel_index, creator, schedule_file_name, voltage_lim_low/high, cycle_mode, test_ID). Mine those for the full field set; they can sit in TestMeta or a sibling CellMeta record (decision deferred — see the design note).

Follow-ups¶

• step_types
• 'charge', 'discharge', 'rest'
• limit to those or allow for others?
• how to deal with 'IR' steps in Arbin or other cycler specific step-types?
• Reference vocabulary now defined as cellpycore.config.StepType (issue #24); it is a non-validating reference set, so other values are still allowed.
• step_type_detail
• how is this is intended to be used
• step_mode
• "CV", "CC", "CP", "None"
• similar to step_type: limit to those, or allow for more?
• Reference vocabulary now defined as cellpycore.config.StepMode (CC/CV/CP); "None" means a null/missing value, not a stored string (issue #24).
• Substep Number
• in our previous notes we listed "substep number"
• do we need this? and if so, what would it be used for?
• epoch_time_utc
• should this be a float or a datetime object? Resolved (issue #32): stored as int64 nanoseconds since the Unix epoch, UTC. Convert to float seconds UTC or a native datetime with the helpers in cellpycore.timestamps.
• cycle_num
• usually provided by the tester
• enough with keeping it, or do we need an updated cycle number?
• cycle_type
• details to be discussed
• Reference vocabulary now defined as cellpycore.config.CycleType (Standard/GITT/ICI/Characterization); may later migrate to test metadata as test_type (issue #24).
• test_family / test_type (now in TestMeta, not raw columns)
• test_family is the broad classification (e.g. "reference capacity test", "rate test"); test_type is the detailed one within a family (e.g. "GITT", "current interrupt").
• limit to a controlled vocabulary, or allow free text? (same open question as cycle_type / step_mode)
• channel_status — Resolved (issue #10): removed. Not part of the harmonized raw format.
• source_uuid
• does this contain info about the channel?
• if not, should we add a channel number/identifyer? (string)
• test_id / merging
• test_id is the compact per-row key that lets a single object hold many merged test files; downstream grouping must use composite keys (test_id, cycle_num, step_num, ...) because cycle_num/step_num collide across files.
• test-level descriptors (test_family, test_type, mass, nominal capacity, cycle_mode, ...) should live in a normalized per-test metadata table keyed by test_id rather than be repeated on every raw row. See .issueflows/04-designs-and-guides/test-metadata-and-merging.md.

Other discussion points¶

• Units
• Units (based on Rasmus' suggestion) are included in the header table above
• EIS, cyclic voltametry, 3-electrode setup
• Data resolution (for definition of step types)