p2 — Blistering-Fast S3-Compatible Object Storage

Key Features

🔌

S3 API Compatible

Drop-in replacement for S3. Works with AWS SDKs, MinIO warp, Cyberduck, and any standard REST tooling.

⚡

Microsecond Cryptography

MD5 and SHA256 hashing offloaded to p2_s3_crypto, a custom Rust extension that releases the Python GIL.

📡

Zero-Copy Reads

GET operations use Nginx X-Accel-Redirect to serve files via kernel-level sendfile(), bypassing Python entirely.

🚀

Framework-less Writes

PUT traffic is intercepted at the raw ASGI layer before Django loads — matching compiled Rust backend performance.

🗄️

LMDB Metadata

Petabyte-scale metadata stored in LMDB with lock-free reads, batched writes, and zero-copy memory-mapped access.

🎛️

Control Plane UI

Modern management interface with Django Ninja, fully typed OpenAPI docs, bucket policies, CORS, ACLs, and lifecycle rules.

Architecture

Control Plane

Django Ninja + Pydantic

Account provisioning & API keys
Bucket creation, CORS, ACLs
Bucket policies & lifecycle rules
Multipart upload orchestration
Presigned URL generation
Web UI & OpenAPI docs

→

S3 traffic detected

→

Data Plane

Raw ASGI + Rust Extensions

ASGI interceptor steals raw bytes off Granian sockets
Bypasses Django middleware completely
p2_s3_crypto computes hashes without stalling the event loop
Payload streamed directly to disk
Metadata pushed to LMDB via batched async queue
Nginx sendfile() for zero-copy GET responses

Benchmarks

MinIO warp tool — 4 KiB objects, 20 concurrent connections, 30s duration, localhost, no TLS.

Operation	Metric	p2	RustFS
GET	Avg Throughput	23.56 MiB/s	19.22 MiB/s
	Avg Obj/s	6,031	4,922
	p50 Latency	3.0 ms	3.8 ms
	p99 Latency	14.8 ms	18.1 ms
PUT	Avg Throughput	5.05 MiB/s	10.11 MiB/s
	Avg Obj/s	1,293	2,588
	p50 Latency	18.5 ms	8.2 ms
	p99 Latency	56.8 ms	22.2 ms

p2 delivers 22% faster GET throughput than RustFS thanks to Nginx zero-copy sendfile(). PUT throughput trades off against Python's write path — the raw ASGI interceptor + Rust hashing keeps it competitive at 1,293 obj/s.

p2 vs RustFS warp benchmark terminal output

Side-by-side warp benchmark: p2 (left) vs RustFS (right)

Getting Started

🐧 Native Linux

bash scripts/run_without_docker.sh

Review or create .env before first run
Install and run Dragonfly or another Redis-compatible server first
Auto-installs Python dependencies
Binds Granian to localhost:8787
Generates the host Nginx config for your local storage path
Set P2_DEBUG=true in .env for verbose logs
~586 MiB RAM across 4 Granian workers at idle

🐳 Docker

bash scripts/setup.sh

Use this for the first Docker setup
Creates or updates .env, installs Nginx config, and starts the stack
Web server on localhost:8787
Includes Dragonfly/Redis-compatible broker + arq workers
Storage auto-mounted at ./storage/
After setup, manage containers with normal docker compose commands

After First Docker Setup

docker compose up -d
docker compose down
docker compose logs -f web

Nginx Zero-Copy Setup

For maximum GET performance, pair with Nginx:

# .env
P2_STORAGE__USE_X_ACCEL_REDIRECT=true

Point Nginx at http://127.0.0.1:8787 (Granian). The setup scripts generate the host config automatically.

Use with AWS CLI

aws configure set default.s3.signature_version s3v4
aws --endpoint-url http://localhost s3 mb s3://my-bucket
aws --endpoint-url http://localhost s3 cp file.txt s3://my-bucket/
aws --endpoint-url http://localhost s3 ls s3://my-bucket/
aws --endpoint-url http://localhost s3 presign s3://my-bucket/file.txt

Test Suite

57 tests — all passing. Run with: .venv/bin/python manage.py test p2.s3.tests.test_unit p2.s3.tests.test_integration

Unit Tests (35)

Presigned Tokens — roundtrip, wrong bucket/key/method, expired, tampered signature, malformed, max expiry clamping
CORS — origin matching, wildcard/subdomain, rule matching, header application, XML parse/build roundtrip
Bucket Policy Engine — parse valid/invalid JSON, allow/deny evaluation, deny-overrides-allow, wildcard actions, resource matching
AWS SigV4 Auth — query string parsing, header parsing, missing params, HMAC key derivation, signing key determinism
Error Classes — HTTP status codes and S3 error code strings

Integration Tests (22)

Bucket Ops — head, delete, versioning, location, list empty, ACL get/put, CORS put/get, policy CRUD, lifecycle CRUD, invalid policy rejection
Object Lifecycle — PUT with ETag/MD5 verification, HEAD (content-type, size, etag), copy object, list after put, presigned URL generation, delete
Object Not Found — head 404, get 404, no-such-bucket, tagging 404, ACL 404, copy missing source, multi-delete nonexistent
Authentication — invalid credentials rejected, wrong secret key → SignatureDoesNotMatch, multi-user isolation (cross-bucket access denied)