P1SCode — Architecture (language & SDK)

This page describes how the P1SCode stack is put together: which components exist, what they own, and how data flows from .p1s to binaries. It pairs with P1SCode — Modules, imports, and the std story, P1SCode — Language reference (concise), and P1SCode Toolchain Workflow.

1. Design intent

Goal

Today’s approach

One parse/lex contract

Shared crate SDK/p1sdk/p1sc_syntax: lexer, import_resolve, parser, AST, parse_program_with_imports.

Shippable programs

Host Python driver SDK/p1sdk/p1sc.py: merge → tokenize → transpile → emit Rust into apps/p1scode_app (or template path) → Cargo.

Fast feedback on image

apps/p1sc → /bin/p1sc.elf: same p1sc_syntax merge + lex + parse; no transpiler on device.

Truth in tests

SDK/p1sdk/fixtures/ + run_fixtures.py; cargo test on p1sc_syntax (host triple).

Code generation is intentionally not duplicated in Rust for M1: the transpiler remains the single emitter for production ELF.

2. Layer diagram

        flowchart TB
  subgraph sources["Sources"]
    P1S[".p1s files"]
  end

  subgraph merge["Import expansion"]
    PY_MERGE["p1sc.py resolve_imports"]
    RS_MERGE["p1sc_syntax::import_resolve"]
  end

  subgraph lexparse["Lex + parse"]
    LEX["lexer"]
    PARSE["parser → AST"]
  end

  subgraph host_only["Host only"]
    TOK_PY["p1sc_lexer_unified / tokenize"]
    TRANS["transpile → Rust source"]
    CARGO["Cargo → ELF"]
  end

  subgraph device["On device"]
    P1SC_ELF["p1sc.elf check"]
  end

  P1S --> PY_MERGE
  P1S --> RS_MERGE
  PY_MERGE --> TOK_PY
  TOK_PY --> TRANS
  TRANS --> CARGO
  RS_MERGE --> LEX
  LEX --> PARSE
  PARSE --> P1SC_ELF

Important: Merge is implemented twice (Python and Rust) with intentionally aligned behavior; it is verified by fixture lex parity, parse_program_with_imports tests, and import_resolve unit tests—not by calling Rust from Python in the driver.

3. Component inventory

Component

Path

Responsibility

Profile / keywords

SDK/p1sdk/p1scode_profile.json

Keyword set shared with lexer::KEYWORDS.

Python driver

SDK/p1sdk/p1sc.py

CLI, diagnostics JSON shape, merge, tokenize, transpile, build orchestration.

Unified lexer (Python)

SDK/p1sdk/p1sc_lexer_unified.py

Rust-aligned tokens for transpile.

Rust syntax crate

SDK/p1sdk/p1sc_syntax/

Lexer, parser, AST, import_resolve, parse_program, parse_program_with_imports, host tests, dump_tokens.

Native checker

apps/p1sc/

p1sc.elf: syscall read_file + merge + lex + parse; --import-root, --json-diagnostics.

Diagnostics bridge

SDK/p1sdk/p1sc_diag_bridge.py

Wraps p1sc.py --json-diagnostics for editors (JSON / LSP shapes).

Fixture harness

SDK/p1sdk/run_fixtures.py

Python success/error/bridge cases + p1sc_syntax tests + optional Rust↔Python lex parity.

App template

SDK/template/p1scode-app/

Starting layout, build.ps1, sample main.p1s + lib/.

4. Data shapes (contract summary)

  1. Merged source (UTF-8 text): result of import graph expansion; file-import lines removed; virtual imports stripped (see P1SCode — Modules, imports, and the std story).

  2. Token stream (Python): tuples (KIND, lexeme) with STRING, KEYWORD, IDENTIFIER, SYMBOL.

  3. Rust tokens: richer TokenType; dump_tokens can emit JSON comparable to Python for parity.

  4. Diagnostics (host + native check): JSON array of { code, message, file, line, column } — see P1SCode Toolchain Workflow.

5. Evolution directions (architecture)

These are roadmap items, not promises:

  • Single merge implementation on host: e.g. subprocess or extension to dump_tokens / small p1sc_front crate—reduces drift risk.

  • Transpile from AST (Rust or Python): today transpile is token-driven; moving to AST → Rust would tighten parser/transpiler coupling.

  • Typed IR or MIR: later layer for optimization and native codegen without Cargo on device.

  • Versioned language / std: tie p1scode_profile.json and module namespaces to a semver or edition flag.

For M1-style milestones and ramdisk layout, see newdocs/p1scode-m1.md.

See also