sparkles:dmd-lsp — Feature Requirements
Status: in design/implementation · Date: 2026-07-30 · Scope: the requirement inventory for the D-native twoslash backend; overview and milestones in the index.
Status legend and ID conventions: hue spec. Port provenance: VisualD's vdc/dmdserver/ (Boost-1.0) — semvisitor.d, semanalysis.d, dmdinit.d, dmderrors.d; the COM shell dmdserver.d is discarded, not ported.
Build & pinning (BLD)
The DMD frontend is consumed as a dub git dependency on a patched branch of the user-owned fork, so plain dub build/dub test work identically in the devshell, on bare machines, and (offline, via the lock) under Nix.
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| BLD1 | A dmdserver-dub branch on PetarKirov/dmd must make the fork dub-consumable: rainers/dmd@dmdserver (3023f50fcf) + versions "LanguageServer" in all four library subpackages (the patch touches files in each) + Linux/dub build fixes (glue/* exclusion; typed asRange keys). Pinned states are tagged (dmdserver-dub/<VERSION>+ls.N); the branch is never force-pushed. | partial | fork branch dmdserver-dub (local); plan §Fork |
| BLD2 | sparkles:dmd-lsp must pin the fork by SHA in libs/dmd-lsp/dub.sdl (repository="git+…" version="<sha>") and dub.selections.json, with a matching git entry ({version, repository, sha256}) in nix/dub-lock.json so buildDubPackage builds offline. | full (acef0edd) | plan §Track B/7 |
| BLD3 | Semantic analysis of samples importing object/std needs frontend-matched runtime sources: a dmd-import-paths nix package (fork druntime/src derived from the dub-lock rev + dlang/phobos at the fork's VERSION tag), exported as SPARKLES_DMD_IMPORT_PATH (devshell + wrapProgram --set-default). | full (acef0edd) | plan §Track B/7; ts-grammars/SPARKLES_TS_GRAMMAR_PATH precedent |
| BLD4 | The frontend must build with the project toolchain — both pkgs.ldc and pkgs.dmd hosts — with no -preview=in/-preview=dip1000 forced onto frontend sources (sparkles:dmd-lsp deliberately omits the repo-standard preview dflags; documented in its dub.sdl). | partial | S1 spike: ldc2 libdmd_frontend.a green |
| BLD5 | twoslash-extract must be a flake package whose wrapper carries everything analysis needs at runtime — SPARKLES_DMD_IMPORT_PATH (--set-default, so an exported one still wins) plus dub and a D compiler on PATH for --dub — and .#hue must point SPARKLES_TWOSLASH_EXTRACT at it, so live D types work from nix run .#hue with nothing exported. Build-time: git (the fork's config.d pre-generate step execs git describe). | full | nix/packages/twoslash-extract.nix; nix/packages/hue.nix |
| BLD6 | Anything linking the frontend builds as checked (optimize + inline + debugInfo, deliberately no releaseMode): the frontend leans on its own asserts, and -release turns the LanguageServer Expression.copy() check into silent undefined behaviour. Measured on live_types.d --dub: debug 2.00 s, checked 1.09 s, release 1.06 s per analysis — the checks cost ~3%. The flake twoslash-extract selects it through buildSparklesApp's dubBuildType parameter (default release); the fixup strip drops the debug info again, so it costs 0.1 MiB of binary (6.6 against 6.5) and nothing in the closure, and assert messages carry their own file:line. | full | buildType "checked" in dmd-lsp / twoslash-d / twoslash-extract; nix/packages/twoslash-extract.nix |
Semantic core (COR)
libs/dmd-lsp — configure, analyze once, query.
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| COR1 | One full semantic pass per sample (parse → importAll → dsymbolSemantic → deferred → semantic2 → semantic3) must answer all queries — diagnostics, tips, identifier spans — with no re-analysis (the four-query contract of issue #120 §4 / hue DMD1). | full (ec71308d) | port of semanalysis.analyzeModule |
| COR2 | The Analyzer is single-use: one analyze() per process-level dmdInit; a second call asserts. The dmdinit.d mangled-name dmdStatics reset table and dmdReinit() are omitted — batch isolation comes from one-analysis-per-process (EXT2); re-analysis support (mainline's deinitializeDMD() path) is deferred follow-up work. | full (ec71308d) | plan §Track B/8; issue #124 known-costs |
| COR3 | Diagnostics must be captured structured via the frontend's diagnosticHandler hook — Diagnostic{filename, startLine/Col, endLine/Col, kind (error/warning/deprecation/message), message} with supplemental notes attached — never parsed back out of rendered strings. errorLimit is uncapped; in-sample vs imported-module diagnostics are distinguishable by filename. | full (ec71308d) | port of dmderrors.d |
| COR4 | The public API speaks DMD's native coordinates — 1-based line, 1-based UTF-8-code-unit column — and nothing else; byte-offset conversion is the analyzer's job (NTN3), so the core stays twoslash-free. | full (032f3b35) | api facade design |
| COR5 | dmdSetupParams must target the host OS (the dmdserver original hardcodes Target.OS.Windows), and accept import dirs, string-import dirs, version idents, and a parsed -preview/-dflags subset (@dflags: feeds this). | full (ec71308d) | port of dmdinit.dmdSetupParams |
| COR6 | Imports must resolve against Options.importPaths (defaulted from SPARKLES_DMD_IMPORT_PATH), with the in-memory sample module served by Module.loadModuleHandler first; environment-dependent tests skip (never fail, never silently pass) when the variable is unset. | full (ec71308d) | port of semanalysis.loadModuleHandler; sparkles.test_runner.skip |
Type oracle (TIP)
The semvisitor.d port — near-verbatim, per issue #124 ("the crown jewel").
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| TIP1 | tipAt(line, col) must return the resolved/inferred type of the innermost node covering the position — arbitrary sub-expressions, not just declarations — as Tip{kind, code, doc} (the (kind) \code`shape), readingExpression.typepopulated by semantic and the fork'sresolvedTo back-pointers. Lowered/generated code never surfaces (loweredLoc, needsCodegen=false`). | full (032f3b35) | semvisitor.findTip/tipDataForObject |
| TIP2 | identifierSpans() must classify every identifier in the module (kind + positions per identifier) from the same pass — the source of hover-per-identifier nodes and, later, semantic-token highlighting (hue SEM1). | full (032f3b35) | semvisitor.findIdentifierTypes |
| TIP3 | definitionAt(line, col) (goto-definition) ports with the visitor and is exposed on the facade — not consumed by twoslash v1, but free in the near-verbatim port and needed by the D4 server. | full (032f3b35) | semvisitor.findDefinition |
| TIP4 | Completions (findExpansions / hue NOT2) are full (32520ca0) (issue #124 D3); the facade carries a stub so the seam is stable. | deferred | issue #124 D3 |
| TIP5 | A function tip must carry its signature's structure, not only its text: where it may break (the runtime parameter list before the template list), which runs abbreviate (nested template arguments, module prefixes) and to what, which spans are the four effect attributes, and its in/out contracts and template constraint. The structure is printed from the AST alongside the text — never scanned back out of it, since the two hdrgen frames put attributes on opposite sides and pure can occur inside a default argument. text stays byte-identical to what hdrgen produced, so every offset indexes it. @system on a variable survives too, which the type-only variable path used to drop. | full | signature.renderSignature; signature.matchesHdrgenOnRealSources |
| TIP6 | The oracle answers for any module the analysis covers, not only the root input: identifierSpans/tipAt/definitionAt take the module as a parameter (looked up by filename among the analyzed set), and the facade can enumerate that set. One analyze() of a.d already produced semantic modules for everything a imports — refusing to query them would force a second process per file for no analytical reason. Backs EXT8. | not started | proposed AnalyzedModule set + by-filename lookup on the facade |
DDoc extraction & rendering (DOC)
The D analog of TypeScript twoslash's JSDoc extraction: documented symbols carry their doc comment into the rendered overlay.
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| DOC1 | The core must retain and surface ddoc: doc comments stay attached through semantic (the fork keeps them on; dmdserver forces ddoc.doOutput), and tipAt returns the resolved symbol's doc comment as Tip.doc — for hovers and ^? queries alike. | full (032f3b35) | semvisitor.tipDataForObject (TipData.doc) |
| DOC2 | The analyzer must populate Node.docs from Tip.doc on hover/query nodes so the shipped renderers display it: the HTML popup docs block (markdown-rendered via TwoslashHtmlOptions.renderDocsMarkdown, hue TWH4), and the ANSI/GUI surfaces. A ddoc-bearing sample is part of the golden fixture corpus. | full (618e98a0) | plan §Track B/10; render_html.d |
| DOC3 | Well-known ddoc sections (Params:, Returns:, See_Also:, Deprecated:, …) map onto Node.tags [name, text] pairs — the analog of JSDoc @param chips (hue TWH3). v1 may pass the doc body through with ddoc macros untranslated (documented caveat; see DDoc guidelines); full section/macro handling is a refinement. | full (d7a33164 + 49da12f5; matrix: ddoc.md) | reference TS fixture 15-markdown-docs |
| DOC4 | Documentation must follow the symbol, not the file: hovering an import, a selectively-imported name, or a call site in another module surfaces that declaration's ddoc. Two indirections are resolved — DMD loads imports with doDocComment = 0 (the Analyzer's loadModuleHandler overrides it, or nothing outside the root module has a comment at all), and an eponymous member's docs are split between it and its TemplateDeclaration, which are merged the way dmd.doc.emitComment merges them. | full | api.Analyzer.analyze; visitor.docForSymbol; visitor.docForSymbol.importedSymbolsCarryTheirDocs |
Analyzer (NTN)
sparkles:twoslash-d — notation in, nodes out. The marker grammar inventory is owned by hue/twoslash.md NOT1–NOT8; these rows pin the pipeline behavior. v1 markers: ^?, the ---cut--- family, @errors: / @noErrors, @dflags: / @import:, custom tags. Deferred: ^| (TIP4), ^^^, @filename:, @dub:.
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| NTN1 | The notation parser is pure text → markers (no compiler dependency): markers point at the line above and align by caret column; notation lines are stripped from the display code; cut regions stay in the compiled source but leave the display. @errors:/@noErrors parse into verification metadata (consumed by EXT3/L12) — extraction always emits every diagnostic as a node. | full (f8c9a846) | plan §Track A/6; TS regexp.ts port |
| NTN2 | Positions are two-phase: nodes are built with start/length in full-source bytes, remapped through the cut map (nodes wholly inside removed ranges are dropped with a warning), then line/character resolve against the post-cut display code. | full (618e98a0) | TS removeCodeRanges/resolveNodePositions |
| NTN3 | Byte ↔ line/col conversion happens only at the dmd-lsp seam, via sparkles.base.text.lineindex (LineIndex.offsetOfDmd for DMD's 1-based UTF-8-code-unit columns). v1 character is the byte column of start (exact for ASCII; display-width refinement deferred). | full (618e98a0) | plan §Track A/2 |
| NTN4 | The emitted payload declares itself: language: "d" and offsetEncoding: "utf-8" on TwoslashReturn (both @WireOptional, absent ⇒ the TypeScript legacy: "typescript"/UTF-16). Ingest converts UTF-16 offsets only for legacy payloads — a D payload round-trips through loadTwoslashFile byte-exact, including non-ASCII. Encoding the existing TS fixtures stays byte-identical. | full (a0626e09) | plan §Track A/3 |
Extractor (EXT)
apps/twoslash-extract — the batch CLI.
| ID | Requirement | Status | Traces to |
|---|---|---|---|
| EXT1 | twoslash-extract <sample.d> writes <sample>.twoslash.json beside the input (or --out <path|dir>); --import (repeatable) prepends to $SPARKLES_DMD_IMPORT_PATH; --dflags merges with in-sample // @dflags:. | full (5aa94285) | plan §Track B/11 |
| EXT2 | One analysis per process: directory mode re-execs the binary per file, so DMD's global mutable state is never reused across samples (the isolation guarantee behind COR2). | full (5aa94285) | plan §Track B/11 |
| EXT3 | --verify re-extracts and diffs against the existing payload (exit 1 on drift) — the golden-fixture guard, and the entry point apps/ci twoslash verification (L12) builds on, where @errors: patterns match by message/{{_}} glob (D has no stable numeric codes; Node.code stays 0, id is synthetic). | partial (5aa94285; ci wiring + {{_}} globs = L12) | hue NOT5; apps/ci {{_}} |
| EXT4 | A committed D sample + fixture corpus lives in libs/twoslash-d/examples/{src,fixtures} (mirroring the TS corpus in libs/twoslash/examples), regenerated only as deliberate golden-snapshot commits pinned to the fork SHA. | full (67c04784) | plan §Track B/10 |
| EXT6 | --dub analyzes an input in its enclosing dub project's context (import paths, string-import paths, version/debug identifiers, dflags via dub describe), with --dub-config/--dub-build selecting a configuration or build type; off by default so the in-repo golden corpus is unaffected. Owned by PRJ10/PRJ11. | full (59e623ff) | project context |
| EXT5 | hue renders D payloads as D: the highlight language and the popup-signature re-highlight language come from TwoslashReturn.effectiveLanguage (today both hardcode "typescript": overlay.popupLanguage, app.d's highlightInjected). TS fixtures keep rendering exactly as before. | full (1344c42b) | plan §Track A/4–5 |
| EXT7 | --serve turns the extractor into a resident oracle: the lazy payload on stdout line 1, then JSON lines — {"tip": <nodeIndex>} → {"node", "text", "docs", "tags", "signature"} — until stdin closes. A malformed request answers {"error": …} and the loop continues. The reply carries the same signature structure (TIP5) the batch payload puts on the node, so a live-resolved hover reflows and abbreviates identically; a reply without it still resolves and renders flat. --unittest analyzes with -unittest (the unittest version identifier, not just useUnitTests), which is what a version (unittest) block needs to exist at all. | full | app.runServe; analyze.wireSignature; hue LIV7 |
| EXT8 | The oracle serves its whole analyzed set, not one file. One --serve analysis already covers every module its input transitively imports, so the protocol must expose that: a {"files": true} request answers the list of analyzed module paths, and {"payload": "<path>"} / {"tip": <nodeIndex>, "file": "<path>"} answer the lazy payload and tips for any file in the set (node indices are per file). A path outside the set answers {"error": …} so the client knows to spawn its own analysis rather than guess. This is what lets a viewer move between files of one project — and a diff viewer between files of one side — without a process per file (hue LIV8). | not started | proposed --serve extension over EXT7; AnalyzedModule lookup by filename |
| EXT9 | API-surface digest: a {"digest": "<path>"} request answers that module's declared symbols with their resolved signatures, attributes and effects — a stable, order-independent summary cheap enough to request for every changed file in a review. Comparing two revisions' digests yields the per-file semantic verdict hue's diff badges render (signatures changed / implementation only / doc only / type-preserving, DVT4); it is deliberately coarser than a per-identifier comparison, which stays on-demand. | not started | proposed digest emitter over the TIP oracle; hue DVT4 |
Non-goals (v1)
- Re-analysis in one process (
dmdReinit/dmdStatics) — seeCOR2. - Completions, references, semantic-token overlay — issue #124 D3.
- A JSON-RPC LSP server — issue #124 D4; the facade is shaped so the server grows around the core, not a rewrite.
- The
pragma(msg)fallback backend (hueDMD4) — the seam allows it; nothing v1 ships needs it. - Multi-file samples (
@filename:) and dub-dependency injection (@dub:) — deferred with their markers.