Overview
Playwright Automation Dashboard is an enterprise QA automation monitoring platform that provides real-time visibility into test execution across multiple CI/CD pipelines. It features a React frontend with a Flask API backend, JWT-based authentication, live test progress streaming, visual regression comparison with pixel-level diff highlighting, and historical trend analysis for test reliability metrics.
Problem Statement
QA teams running hundreds of Playwright tests across multiple environments struggle with visibility. CI/CD logs are noisy, failures are hard to triage, and visual regressions are caught too late. Teams needed a centralized dashboard that shows test execution in real time, highlights visual differences clearly, and provides actionable failure analysis without context-switching between tools.
Architecture Overview
The platform uses a React SPA frontend communicating with a Flask backend via REST endpoints secured with JWT authentication. TypeScript powers the frontend for type safety, while Python handles test result processing and storage. The system integrates with CI/CD webhooks for automatic test run ingestion and provides WebSocket-based real-time updates for active test runs.
Tech Stack
Key Features
Real-Time Test Execution Viewer
Live feed showing test progress, pass/fail status, and execution time as tests run across pipelines
JWT Authentication
Secure role-based access control with token refresh and session management for team collaboration
Visual Regression Comparison
Side-by-side screenshot comparison with pixel-level diff highlighting and configurable sensitivity thresholds
Failure Analysis
Automatic error categorization, flaky test detection, and historical reliability scoring per test
Screenshots
Engineering Challenges
Real-time updates at scale with hundreds of concurrent test runs
Implemented a pub/sub pattern with room-based WebSocket channels, allowing clients to subscribe only to relevant test runs. Added backpressure handling and message batching for high-throughput scenarios.
JWT token management with secure refresh flow
Built a dual-token system with short-lived access tokens and longer-lived refresh tokens stored in httpOnly cookies. Automatic token refresh on 401 responses with request queuing during refresh.
Correlating test results across parallel CI pipelines
Designed a run-grouping system using commit SHA and pipeline metadata to correlate results from sharded test executions into unified views.
// JWT refresh interceptor for API requests
api.interceptors.response.use(
(response) => response,
async (error) => {
if (error.response?.status === 401 && !error.config._retry) {
error.config._retry = true;
const newToken = await refreshAccessToken();
error.config.headers.Authorization = `Bearer ${newToken}`;
return api(error.config);
}
return Promise.reject(error);
}
);@app.route('/api/runs', methods=['POST'])
@jwt_required()
def create_test_run():
data = request.get_json()
run = TestRun(
name=data['name'],
pipeline_id=data.get('pipeline_id'),
commit_sha=data.get('commit_sha'),
status='pending',
created_by=get_jwt_identity()
)
db.session.add(run)
db.session.commit()
return jsonify(run.to_dict()), 201Testing Strategy
The platform is tested using Playwright itself (meta-testing). E2E tests cover the full flow from webhook ingestion through dashboard rendering. Unit tests validate the screenshot diff engine with known image pairs. Integration tests verify API endpoints, JWT authentication flows, and database consistency under concurrent writes.
Deployment
The Flask backend is deployed as a containerized service with the React SPA served separately. JWT secrets are managed through environment configuration. The system supports horizontal scaling behind a load balancer with sticky sessions for WebSocket connections.
Lessons Learned
- JWT refresh token rotation prevents replay attacks but requires careful handling of concurrent requests during refresh
- Visual regression thresholds need to be configurable per-project because acceptable differences vary widely
- Dogfooding your own testing tool reveals UX issues that synthetic test scenarios never surface
- Batching WebSocket messages significantly reduces client CPU usage during high-throughput test runs