Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
f760661
docs: PipeTests framework design spec
muit Sep 3, 2026
f7f26b3
docs: PipeTests implementation plan
muit Sep 3, 2026
31a9b45
build: add PipeTests library target
muit Sep 3, 2026
683e3f6
feat: declare PipeTests registration API
muit Sep 3, 2026
cbe2f17
feat: add PipeTests registry and runner
muit Sep 3, 2026
85ccb14
test: add PipeTests self-test suite
muit Sep 4, 2026
0c3bd53
docs: macro-free registration, PipeTestsLib rename in plan+spec
muit Sep 4, 2026
cddd073
feat: add Expect fluent matcher
muit Sep 4, 2026
e4bb22b
fix: use source_location for call-site failure reporting
muit Sep 4, 2026
20136dd
docs: source_location for call-site EXPECT reporting
muit Sep 4, 2026
832d97c
refactor: TestSettings RunTests, Pipe types, TestDescribe/TestContext…
muit Sep 4, 2026
ebc9b5a
test: file-scope auto-register, PipeTest lib, PipeTests suite, TestSe…
muit Sep 4, 2026
7ced119
test: clean up spec files, remove PipeTestsSelf, add CLI features
muit Sep 4, 2026
ab5ffb0
fix: missing period in PipeTests colored summary format string
muit Sep 4, 2026
f156ef1
fix: correct placeholder count in colored summary format string
muit Sep 4, 2026
e06836f
Added log colors and docs
muit Sep 4, 2026
b5ab97f
Added new reporters, fixes, help and version and formatting
muit Sep 4, 2026
7e540b0
P_SPEC macro improvements, added TTypeId
muit Sep 5, 2026
b480ebc
Refactored Reflection and Type headers
muit Sep 6, 2026
2808c5f
Solved dependency issue
muit Sep 6, 2026
dc14d8e
Small fix for Windows
muit Sep 6, 2026
3d27ec3
SMall fix on non-windows builds
muit Sep 6, 2026
f5ccde2
Removed plans
muit Sep 6, 2026
d0d1c0f
Small changes to testing api
muit Sep 6, 2026
972a21d
Added custom reporters and documentation
muit Sep 6, 2026
83bfc71
Added new line on PipeType
muit Sep 7, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 0 additions & 14 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,6 @@ pipe_target_define_platform(Pipe)
target_include_directories(Pipe PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/Include>)
target_include_directories(Pipe PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/Src>)
file(GLOB_RECURSE PIPE_SOURCE_FILES CONFIGURE_DEPENDS Src/*.cpp Src/*.c)
list(FILTER PIPE_SOURCE_FILES EXCLUDE REGEX ".*/Src/Tests/.*")
target_sources(Pipe PRIVATE ${PIPE_SOURCE_FILES})
target_compile_definitions(Pipe PUBLIC P_VERSION="${PROJECT_VERSION}")
target_compile_definitions(Pipe PRIVATE NOMINMAX)
Expand All @@ -83,19 +82,6 @@ pipe_target_shared_output_directory(Pipe)
pipe_target_disable_rtti(Pipe PRIVATE)


################################################################################
# PipeTest (test framework library, not part of the runtime Pipe library)

add_library(PipeTest STATIC Src/Tests/PipeTest.cpp)
add_library(Pipe::Test ALIAS PipeTest)
pipe_target_define_platform(PipeTest)
target_include_directories(PipeTest PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/Include>)
pipe_target_enable_CPP20(PipeTest)
pipe_target_disable_rtti(PipeTest PRIVATE)
pipe_target_shared_output_directory(PipeTest)
target_link_libraries(PipeTest PUBLIC Pipe)


################################################################################
# Pipe Tests (compiled) executable

Expand Down
234 changes: 234 additions & 0 deletions Docs/PipeTest.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,234 @@
---
title: Pipe Test
---
**Header:** [`<PipeTest.h>`](https://github.com/PipeRift/pipe/blob/main/Include/PipeTest.h)`]
**Namespace:** `p`
**Uses:** [`PipeReflect`](./PipeECS.md) `PipeStrings` `PipeTime`

## Overview

`PipeTest` is test framework built into Pipe, used by the library to test itself.
It is heavily inspired by `Bandit`, following similar APIs and outputs to fully support existing IDE tools built for that library.

```cpp
#include <PipeTest.h>

using namespace p;

P_SPEC("Math", []()
{
BeforeEach([]() { /* runs before each test */ });

It("clamps to range", []()
{
Expect(Clamp(5, 0, 3)).ToEqual(3);
});

XIt("disabled case", []()
{
// never runs
});

AfterEach([]() { /* runs after each test */ });
});
```

## Philosophy

A few guiding decisions shape the whole test API.

- **Macro-free registration:** Registration of tests (`Describe`, `It`, `XIt`, `BeforeEach`, `AfterEach`) are plain functions. The only macro is `P_SPEC`, which it is just syntax sugar to statically register an spec.
- **Immediate global context:** All registration functions live in namespace `p` and operate on a global `TestContext`.
- **Fluent assertions:** `Expect(x)` followed by `ToEqual(y)`, `ToBeTrue()`, `ToContain(sub)`, etc to cover many different testing scenarios.
- **Extensible value formatting:** Failure messages are messaged in a human-readable way, and extensible by the user for new types.
- **Mirror bandit's vocabulary:** `Describe`/`It`/`XIt`/`BeforeEach`/`AfterEach`, `--only`/`--skip`/`--reporter`/`--dry-run`, `Spec`/`Dots`/`Singleline`/`Info` reporters. Migrating existing tests is trivial.
- **No exceptions / no RTTI:** Pipe doesn't require or use RTTI. This is the same with PipeTests.

## Writing Tests
On any cpp file in our test module, we can define one or more specs.

```cpp
// Specs register tests inside and optionally adds root Describe("Files.Paths")
P_SPEC("Files.Paths", []()
{
BeforeEach([]() { /* setup */ });

Describe("Join", []()
{
// Describes group all tests inside. They can be nested.
It("joins two segments", []()
{
Expect(Join("a", "b")).ToEqual("a/b");
});

It("joins three segments", []()
{
Expect(Join("a", "b", "c")).ToEqual("a/b/c");
});

XIt("join four segments", []() { /* 'XIt' are disabled tests */ });
});

It("top-level test", []()
{
Expect(true).ToBeTrue();
Expect(false).ToBeFalse();
});

AfterEach ([]() { /* teardown */ });
});
```

> [!Note]
> Because registration runs from a constructor (`SpecAutoRegister`), `P_SPEC` can only appear at namespace scope. Inside a function, register the spec directly with `RegisterSpec(name, callback)`.

## Running Tests

`PipeTest` exposes two runner entry points:

```cpp
int RunTests(const TestSettings& settings);
int RunTests(int argc, char** argv);
```

The second overload parses CLI args into `TestSettings`.

In CLI, simply call your test binary. You can use these args:

| Argument | Effect |
| ------------------------------------- | --------------------------------------------------------------- |
| `--only=<substring>` | Run only describes/its whose full name contains `<substring>` |
| `--skip=<substring>` | Skip describes/its whose full name contains `<substring>` |
| `--break-on-failure` | Stop on the first failing test |
| `--dry-run`, `-l`, `--list` | Report the full tree as SKIPPED, run nothing |
| `--report-timing` | Report per-test and total run durations |
| `-r=<name>`, `--reporter=<name>` | Select reporter by (case-insensitive) name — see [Reporter Name Lookup](#Reporter-Name-Lookup) |
| `--colorizer=off`, `--no-color`, `-c` | Disable colorized output |
| `--version` | Print Pipe version and exit |
| `--help` | Print usage and exit |

## Reporters

Reporters share the same per-test execution flow but format the run their own way:

| Reporter | Style |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spec`<br>(default) | Indented contexts, `- it <name> ... OK` lines, summary block. Compatible with the Bandit VSCode adapter (one tab per indent level — any other indent breaks hierarchy detection). |
| `dots` | One character per test, color-coded, wrapped at a fixed line width. |
| `singleline` | Live overwriting status line on real terminals (`Executed N tests.`). On redirected streams it prints nothing per-test, only the final summary. |
| `info` | Verbose: `begin/end <context>` lines, per-context totals (`{n} total`, `{n} skipped`, `{n} failed`), `[ PASS ]`/`[ FAIL ]`/`-ERROR->` test lines, and a final failures list. |

`--report-timing` adds a colored duration suffix (e.g. ` (0.00012s)` / `(0.4us)`) to every test line and a total-time line at the end in the `info` reporter.

Custom reporters are supported by inheriting `p::ITestReporter`.
They can be called by their name (`--reporter=SpecialReporter` or just `--reporter=Special`):
```cpp
struct SpecialReporter : p::ITestReporter
{
using Super = p::ITestReporter;
P_STRUCT(SpecialReporter)

// ... override the reporter callbacks ...
};
```

## Filters & Behavior

- Tests not matched by `--only`, matched by `--skip`, marked `XIt`, or after a `--break-on-failure` trigger are all reported as **SKIPPED** (bandit semantics). They are not hidden — only excluded from execution.
- Failures are buffered during the run and emitted by the reporter at the end, so they don't interleave with deferred context output.

---

## Reference
### `P_SPEC` Macro
Automatically registers an battery of tests.

```cpp
P_SPEC("Topic", []()
{
// body
});
```

```cpp
#define P_SPEC static const p::SpecAutoRegister P_CAT(_pipeSpecReg_, __COUNTER__)
```
### Registration Functions

All in namespace `p`. Bodies in `P_SPEC` / `Describe` are stored as `TFunction<void()>` (non-owning, called immediately during registration). Test bodies and hooks are stored as `std::function<void()>` (owning) because they run later from `RunTests`.

| Function | Signature | Purpose |
| ------------------ | -------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| `RegisterSpec` | `(StringView name, TFunction<void()> fn)` | Internal: pushes a named first-level describe, runs `fn`, pops. |
| `RegisterSpec` | `(TFunction<void()> fn)` | Runs `fn` against the virtual root describe. |
| `SpecAutoRegister` | ctor `(StringView, TFunction<void()>)` / `(TFunction<void()>)` | Static-init registrar used by `P_SPEC`. |
| `Describe` | `(StringView name, TFunction<void()> fn)` | Push a nested describe. Runtime-checked inside a `Spec`; logs + ignores otherwise. |
| `It` | `(StringView name, std::function<void()> fn)` | Register a runnable leaf test. |
| `XIt` | `(StringView name, std::function<void()> fn)` | Register a leaf test that is never run. |
| `BeforeEach` | `(std::function<void()> fn)` | Attach a setup hook to the current describe. |
| `AfterEach` | `(std::function<void()> fn)` | Attach a teardown hook to the current describe. |

### `Expect(value)` — Fluent Matchers

`Expect(value, loc = std::source_location::current())` returns an `ExpectValue<T>` bound to the caller's `file:line`. The matchers available today:

| Matcher | Signature | Failure message |
|---------|-----------|-----------------|
| `ToEqual(expected)` | `template<typename E> void ToEqual(const E&) const` | `Expected {actual} to equal {expected}` |
| `ToNotEqual(expected)` | `template<typename E> void ToNotEqual(const E&) const` | `Expected {actual} to not equal {expected}` |
| `ToBeLess(other)` | `void ToBeLess(const Actual&) const` | `Expected {actual} to be less than {other}` |
| `ToBeLessOrEqual(other)` | `void ToBeLessOrEqual(const Actual&) const` | `Expected {actual} to be less or equal to {other}` |
| `ToBeGreater(other)` | `void ToBeGreater(const Actual&) const` | `Expected {actual} to be greater than {other}` |
| `ToBeGreaterOrEqual(other)` | `void ToBeGreaterOrEqual(const Actual&) const` | `Expected {actual} to be greater or equal to {other}` |
| `ToBeTrue()` | `void ToBeTrue() const` | `Expected value to be true` |
| `ToBeFalse()` | `void ToBeFalse() const` | `Expected value to be false` |
| `ToContain(sub)` | `void ToContain(StringView) const` | `Expected {actual} to contain {sub}` |
| `ToNotContain(sub)` | `void ToNotContain(StringView) const` | `Expected {actual} to not contain {sub}` |

- **String-aware equality.** `ToEqual`/`ToNotEqual` treat string-like operands (`StringView`, `const char*`, `std::string`, `p::String`) as strings and compare via `StringView`; everything else uses `operator==`.
- **Containment.** `ToContain`/`ToNotContain` require a `StringView` substring; the actual value is constructed into a `StringView` view for the search.
- **Source location.** The matcher's `file:line` is captured at the call site via `std::source_location` defaults, so failure reports point at the assertion line, not inside the framework.
- **Assertion counter.** Every matcher increments an internal counter. Tests that ran zero asserts are surfaced by reporters as "no assertions" rather than `[ PASS ]`.

## `TestString<T>` — Custom Value Formatting

Failure messages format actual/expected values through `String TestString<T>(const T&)`. The default implementation handles two cases:
- **Formattable types** (`std::formatter<std::remove_cvref_t<T>, char>` exists): `Format("{}", value)`.
- **Non-formattable types** (structs, byte views): a generic placeholder `<value@0x...>` so the framework stays compilable for opaque types.

Predefined specializations and overloads: `bool` (`true`/`false`), `char`, `StringView`, `p::String`, `const char*` (`(null)` for null), `std::string`, and any pointer type (`0xADDR`).

Extend for your own types by specializing the template:

```cpp
template<>
inline String p::TestString<MyType>(const MyType& v)
{
return Format("MyType({})", v.id);
}
```

Prefer `StringView` overloads over `String` so views and literals both work without allocation.

### `TestSettings`

```cpp
struct TestSettings
{
StringView only; // Run only describes/its containing substring.
StringView skip; // Skip describes/its containing substring.
bool dryRun = false; // Report full tree as SKIPPED, run nothing.
bool breakOnFailure = false; // Stop the test run on the first failing test.
bool useColor = true; // Colorized output.
bool reportTiming = false; // Report per-test timing information.
TTypeId<ITestReporter> reporter; // Reporter id; default = SpecReporter.
};
```

Process exit code: `0` when every test passed, `1` otherwise.

### Failure Reporting

- Failures are deferred: per-test assertions only update a per-test buffer (`file:line: msg`); the reporter flushes them at the end of the run.
- Each failed test contributes one entry to the global `failures` list, formatted as `context.test:\n<details>\n`.
- Failure messages are written to **stdout** (not stderr) so test-tooling parsers that follow bandit's convention (e.g. the Bandit VSCode adapter) keep working.
28 changes: 28 additions & 0 deletions Include/Pipe/Core/StringView.h
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

#pragma once

#include "Pipe/Core/Char.h"
#include "Pipe/Core/Hash.h"
#include "Pipe/Core/Optional.h"
#include "PipePlatform.h"
Expand Down Expand Up @@ -84,6 +85,23 @@ namespace p
return str.size() == 1 && str[0] == c;
}

template<typename CharType>
constexpr bool IEquals(const TStringView<CharType> str, const TStringView<CharType> other)
{
return str.size() == other.size()
&& std::equal(str.begin(), str.end(), other.begin(), [](CharType a, CharType b)
{
return TCharHelpers<CharType>::ToLower(a) == TCharHelpers<CharType>::ToLower(b);
});
}

template<typename CharType>
constexpr bool IEquals(const TStringView<CharType> str, const CharType c)
{
return str.size() == 1
&& TCharHelpers<CharType>::ToLower(str[0]) == TCharHelpers<CharType>::ToLower(c);
}

template<typename CharType>
constexpr bool StartsWith(
const TStringView<CharType> str, const TStringView<CharType> subStr)
Expand Down Expand Up @@ -228,6 +246,16 @@ namespace p
return Equals<char>(str, c);
}

P_API inline bool IEquals(const StringView str, const StringView other)
{
return IEquals<char>(str, other);
}

P_API constexpr bool IEquals(const StringView str, const char c)
{
return IEquals<char>(str, c);
}

P_API constexpr bool StartsWith(const StringView str, const StringView subStr)
{
return StartsWith<char>(str, subStr);
Expand Down
50 changes: 36 additions & 14 deletions Include/PipeTest.h
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ namespace p
requires(const T& value) { std::formatter<std::remove_cvref_t<T>, Char>{}; };
} // namespace details

void RegisterSpec(StringView name, TFunction<void()> fn);
void RegisterSpec(TFunction<void()> fn);
P_API void RegisterSpec(StringView name, TFunction<void()> fn);
P_API void RegisterSpec(TFunction<void()> fn);

// Self-registering top-level describe. Registers its spec body on
// construction (same pattern as TTypeAutoRegister). The body runs
Expand Down Expand Up @@ -70,20 +70,42 @@ namespace p
#define P_SPEC static const p::SpecAutoRegister P_CAT(_pipeSpecReg_, __COUNTER__)

// Nested describe. Only valid inside a Spec; otherwise logs an error and ignores.
void Describe(StringView name, TFunction<void()> fn);
P_API void Describe(StringView name, TFunction<void()> fn);
// Register a runnable test in the current describe.
// Bodies are stored until RunTests, so std::function (owning) is required here.
void It(StringView name, std::function<void()> fn);
P_API void It(StringView name, std::function<void()> fn);
// Register a disabled test; never run.
void XIt(StringView name, std::function<void()> fn);
P_API void XIt(StringView name, std::function<void()> fn);
// Setup hook attached to the current describe.
void BeforeEach(std::function<void()> fn);
P_API void BeforeEach(std::function<void()> fn);
// Teardown hook attached to the current describe.
void AfterEach(std::function<void()> fn);
P_API void AfterEach(std::function<void()> fn);

// Reporter interface (defined in Src/Tests/PipeTest.cpp). Forward
// declaration so TestSettings can reference its type id.
struct ITestReporter;

// Reporter interface. Each reporter formats the run differently.
struct P_API ITestReporter
{
P_STRUCT(ITestReporter)

virtual ~ITestReporter() = default;

virtual void TestRunStarting() {}
virtual void TestRunComplete() = 0;
virtual void ContextStarting(StringView) {}
virtual void ContextEnded(StringView) {}
virtual void ItStarting(StringView) {}
virtual void ItSucceeded(StringView) {}
// Test passed but made no assertions (e.g. smoke tests).
virtual void ItSucceededNoAssertions(StringView) {}
virtual void ItFailed(StringView) {}
virtual void ItUnknownError(StringView) {}
virtual void ItSkipped(StringView) {}
};


// Finds a registered reporter type by name (case-insensitive). Matches
// either the full type name ("SpecReporter") or its preffix ("Spec").
P_API TTypeId<ITestReporter> FindReporter(StringView name);

// Settings for a test run.
struct TestSettings
Expand All @@ -100,8 +122,8 @@ namespace p
TTypeId<ITestReporter> reporter;
};

int RunTests(const TestSettings& settings);
int RunTests(int argc, char** argv);
P_API int RunTests(const TestSettings& settings);
P_API int RunTests(int argc, char** argv);


// Extensible value-to-string hook for failure messages.
Expand Down Expand Up @@ -167,10 +189,10 @@ namespace p
namespace details
{
// Format failure message from source location + description.
void Fail(const std::source_location& loc, StringView message);
P_API void Fail(const std::source_location& loc, StringView message);

// Counts an assertion. Used to detect tests that ran no expects.
void CountAssert();
P_API void CountAssert();

// Detect types constructible into a StringView (string-ish values).
template<typename T>
Expand Down
Loading
Loading