This commit is contained in:
Henrik Jess Nielsen
242
scripts/ci/README.md
Normal file
242
scripts/ci/README.md
Normal file
@@ -0,0 +1,242 @@
|
||||
# CI Workflow Scripts
|
||||
|
||||
This directory contains extracted scripts from GitHub Actions CI workflows, organized by workflow type.
|
||||
|
||||
## Overview
|
||||
|
||||
- **Total Scripts**: 41 (27 Bash + 14 PowerShell)
|
||||
- **Documentation**: See `SCRIPT_MAPPING.md` for detailed workflow-to-script mapping
|
||||
- **All Scripts**: Production-ready with proper error handling and documentation
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```text
|
||||
scripts/ci/
|
||||
├── README.md ← This file
|
||||
├── SCRIPT_MAPPING.md ← Detailed workflow-to-script mapping guide
|
||||
├── docker/ ← Docker image build and test scripts
|
||||
├── go/ ← Go bindings scripts
|
||||
├── java/ ← Java bindings scripts
|
||||
├── node/ ← Node/TypeScript NAPI scripts
|
||||
├── python/ ← Python wheel build scripts
|
||||
├── ruby/ ← Ruby gem build scripts
|
||||
├── rust/ ← Rust core and CLI scripts
|
||||
├── csharp/ ← C# bindings scripts
|
||||
└── validate/ ← Validation and linting scripts
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Running a Script
|
||||
|
||||
**Bash scripts:**
|
||||
|
||||
```bash
|
||||
./scripts/ci/docker/build-image.sh core
|
||||
./scripts/ci/python/run-tests.sh true
|
||||
```
|
||||
|
||||
**PowerShell scripts:**
|
||||
|
||||
```powershell
|
||||
& ./scripts/ci/go/build-ffi.ps1
|
||||
& ./scripts/ci/rust/package-cli-windows.ps1 -Target "x86_64-pc-windows-msvc"
|
||||
```
|
||||
|
||||
### Sourcing Scripts
|
||||
|
||||
For library path setup scripts:
|
||||
|
||||
```bash
|
||||
source ./scripts/lib/library-paths.sh
|
||||
setup_all_library_paths
|
||||
./scripts/ci/python/run-tests.sh true
|
||||
```
|
||||
|
||||
## Scripts by Workflow
|
||||
|
||||
### Docker (`docker/`)
|
||||
|
||||
- `free-disk-space.sh` - Clean up CI disk space
|
||||
- `build-image.sh` - Build Docker image variant
|
||||
- `check-image-size.sh` - Validate image size constraints
|
||||
- `save-image.sh` - Save Docker image as tar.gz artifact
|
||||
- `collect-logs.sh` - Collect container logs on failure
|
||||
- `cleanup.sh` - Clean up Docker resources
|
||||
- `summary.sh` - Print test summary
|
||||
|
||||
### Go (`go/`)
|
||||
|
||||
- `build-ffi.sh` - Build FFI library (Unix)
|
||||
- `build-ffi.ps1` - Build FFI library (Windows)
|
||||
- `build-bindings.sh` - Build Go bindings with CGO (Unix)
|
||||
- `build-bindings.ps1` - Build Go bindings with CGO (Windows)
|
||||
- `reorganize-libraries.ps1` - Reorganize FFI libraries for Windows
|
||||
- `run-tests.sh` - Run Go tests with library paths
|
||||
|
||||
### Java (`java/`)
|
||||
|
||||
- `build-java.sh` - Build Java bindings with Maven
|
||||
- `run-tests.sh` - Run Java tests with Maven
|
||||
|
||||
### Node/TypeScript (`node/`)
|
||||
|
||||
- `build-napi.sh` - Build NAPI bindings with artifact collection
|
||||
- `unpack-bindings.sh` - Unpack and install bindings from tarball
|
||||
|
||||
### Python (`python/`)
|
||||
|
||||
- `clean-artifacts.sh` - Clean previous wheel artifacts
|
||||
- `smoke-test-wheel.sh` - Test wheel installation
|
||||
- `install-wheel.sh` - Install platform-specific wheel
|
||||
- `run-tests.sh` - Run tests with optional coverage
|
||||
|
||||
### Ruby (`ruby/`)
|
||||
|
||||
- `install-ruby-deps.sh` - Install bundle dependencies (Unix)
|
||||
- `install-ruby-deps.ps1` - Install bundle dependencies (Windows)
|
||||
- `vendor-kreuzberg-core.py` - Vendor core crate for packaging
|
||||
- `configure-bindgen-windows.ps1` - Configure bindgen headers (Windows)
|
||||
- `configure-tesseract-windows.ps1` - Configure Tesseract (Windows)
|
||||
- `build-gem.sh` - Build Ruby gem
|
||||
- `install-gem.sh` - Install built gem
|
||||
- `compile-extension.sh` - Compile native extension
|
||||
- `run-tests.sh` - Run RSpec tests
|
||||
|
||||
### Rust (`rust/`)
|
||||
|
||||
- `configure-bindgen-windows.ps1` - Configure bindgen headers (Windows)
|
||||
- `run-unit-tests.sh` - Run Rust unit tests
|
||||
- `package-cli-unix.sh` - Package CLI as tar.gz (Unix)
|
||||
- `package-cli-windows.ps1` - Package CLI as zip (Windows)
|
||||
- `test-cli-unix.sh` - Test CLI binary (Unix)
|
||||
- `test-cli-windows.ps1` - Test CLI binary (Windows)
|
||||
|
||||
### C# (`csharp/`)
|
||||
|
||||
- `build-csharp.sh` - Build C# bindings with dotnet
|
||||
- `run-tests.sh` - Run C# tests with dotnet
|
||||
|
||||
### Validate (`validate/`)
|
||||
|
||||
- `run-lint.sh` - Run all linting and validation checks via Task
|
||||
|
||||
## Features
|
||||
|
||||
### Error Handling
|
||||
|
||||
- All Bash scripts use `set -euo pipefail`
|
||||
- All PowerShell scripts use `Set-StrictMode` and error action preferences
|
||||
- Proper exit codes and error messages
|
||||
- Usage information for incorrect arguments
|
||||
|
||||
### Documentation
|
||||
|
||||
- Every script has a descriptive header
|
||||
- Purpose and usage clearly stated
|
||||
- Which CI workflow step uses it
|
||||
- Argument documentation
|
||||
|
||||
### Platform Support
|
||||
|
||||
- Windows-specific operations via PowerShell (.ps1)
|
||||
- Unix operations via Bash (.sh)
|
||||
- Cross-platform scripts detect OS and adjust behavior
|
||||
- Library path setup scripts handle Windows/Linux/macOS
|
||||
|
||||
### Reusability
|
||||
|
||||
- `library-paths.sh` (`scripts/lib/`) - Shared by all workflows for native library configuration
|
||||
- `configure-bindgen-windows.ps1` used by Ruby and Rust
|
||||
- Common patterns consolidated into single scripts
|
||||
|
||||
## Detailed Documentation
|
||||
|
||||
For comprehensive workflow-to-script mapping and usage examples, see `SCRIPT_MAPPING.md`.
|
||||
|
||||
## Usage in Workflows
|
||||
|
||||
### Example: ci-docker.yaml
|
||||
|
||||
**Before (inline commands):**
|
||||
|
||||
```yaml
|
||||
- name: Free up disk space
|
||||
run: |
|
||||
echo "=== Initial disk space ==="
|
||||
df -h /
|
||||
echo "=== Removing unnecessary packages ==="
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
# ... 30+ lines of commands ...
|
||||
```
|
||||
|
||||
**After (using script):**
|
||||
|
||||
```yaml
|
||||
- name: Free up disk space
|
||||
run: ./scripts/ci/docker/free-disk-space.sh
|
||||
```
|
||||
|
||||
### Example: ci-python.yaml
|
||||
|
||||
**Before (inline commands):**
|
||||
|
||||
```yaml
|
||||
- name: Run Python tests
|
||||
run: |
|
||||
cd packages/python
|
||||
if [ "${{ matrix.coverage }}" = "true" ]; then
|
||||
uv run pytest -vv --cov=kreuzberg --cov-report=lcov:coverage.lcov ...
|
||||
else
|
||||
uv run pytest -vv --reruns 1 --reruns-delay 1
|
||||
fi
|
||||
```
|
||||
|
||||
**After (using script):**
|
||||
|
||||
```yaml
|
||||
- name: Run Python tests
|
||||
run: ./scripts/ci/python/run-tests.sh ${{ matrix.coverage }}
|
||||
```
|
||||
|
||||
## Testing Scripts Locally
|
||||
|
||||
You can test scripts locally before running in CI:
|
||||
|
||||
```bash
|
||||
# Test Docker scripts
|
||||
./scripts/ci/docker/free-disk-space.sh
|
||||
|
||||
# Test Python scripts
|
||||
./scripts/ci/python/clean-artifacts.sh
|
||||
./scripts/ci/python/run-tests.sh false
|
||||
|
||||
# Test Rust scripts
|
||||
./scripts/ci/rust/run-unit-tests.sh
|
||||
```
|
||||
|
||||
## Shell Compatibility
|
||||
|
||||
- **Bash scripts**: Compatible with bash 3.2+ (macOS) and bash 4.0+ (Linux)
|
||||
- **PowerShell scripts**: Compatible with PowerShell 5.1+ (Windows) and PowerShell Core 7+ (cross-platform)
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new CI steps or modifying existing ones:
|
||||
|
||||
1. Extract the inline script into a separate file in the appropriate directory
|
||||
2. Add proper error handling (`set -euo pipefail` for bash)
|
||||
3. Include descriptive header comments
|
||||
4. Update `SCRIPT_MAPPING.md` with the new mapping
|
||||
5. Test the script locally before committing
|
||||
|
||||
## Maintenance
|
||||
|
||||
Scripts should be reviewed and updated when:
|
||||
|
||||
- Updating CI workflow logic
|
||||
- Changing build tools or versions
|
||||
- Improving error handling
|
||||
- Adding new platform support
|
||||
|
||||
See each script's header for detailed documentation on its purpose and usage.
|
||||
Reference in New Issue
Block a user