Senior Software Development Engineer in Test (SDET) - Benchmarking

Bitwarden promotes better internet security and safety with the leading open source password management solution for individuals, teams, and business organizations. Our philosophy on security is that “we are all in this together” – we continue to offer a full-featured free version of Bitwarden so that everyone can be protected with strong password management. We also provide enterprise-grade solutions to some of the largest companies in the world, and will continue to innovate in the identity and authentication market as the world starts to grow beyond passwords. Read more on the Bitwarden blog. 

Autofill Coverage owns the measurement of autofill quality end-to-end: establishing a manual baseline across top sites in every major category, automating that baseline into a repeatable regression and benchmarking suite, and reporting coverage, regressions, and competitive standing on a fixed cadence. As a Senior SDET on the Benchmarking side of the Autofill Coverage team, you'll own the automated performance monitoring and benchmarking harness for autofill, localizing regressions to specific merge commits, detecting them on main, and measuring worst-case performance under stress. Autofill already monitors correctness via BIT; you'll build out the three categories that extend it:

• Experience Impact: What autofill costs the page, regardless of whether it fills anything
• Responsiveness: How quickly autofill reacts to what a user does
• Predictability: How stable autofill's behavior is as a whole, run to run and page to page, rather than any single cost or latency figure

When your regression suite finds a mapping defect, you hand it to the Dataset engineers on this team to fix in Map the Web; when it finds anything else, you own routing it to the right team. You'll also work closely with the US Autofill team on triage, instrumentation standards, and release coordination.

WHAT YOU WILL DO

• Design and maintain benchmark scenarios that exercise autofill against busy, mutating, animation-heavy, and deeply-nested pages, using the browser's own performance-timeline tooling rather than active telemetry.
• Build and maintain the underlying test instrumentation, including in-page signals for long-running or delayed rendering work, and custom timing marks the extension emits itself. Some of this data can only be captured in our own controlled test environment, using a debugging connection to the browser that isn't available on a real user's machine. You'll need to understand which signals are which, and design the harness accordingly.
• Own the statistical model behind regression detection.
• Correctly interpret and represent raw engine counters (e.g., heap size, node counts, GC pause) as what they actually are: paired before/after readings, not silently-derived deltas, and know when a signal is meaningful to gate on versus merely reportable.
• Design join keys and event-record schemas so that independent signals (e.g., dropped frames and GC pauses) captured by different subsystems and clocks can be correlated and ordered against each other.
• Build and maintain the harness's read side: consuming a scenario's own declared lifecycle and event marks from the performance timeline, rather than inferring state from rendered DOM output, so runs are comparable and captures remain valid to re-check later.
• Publish results to Datadog, build alerting on benchmark regressions, and help localize a detected regression to the specific merge commit that introduced it.
• Maintain the test-the-web scenario library and its marking conventions, and write Claude skills/documentation so the monitoring strategy stays correctly applied as new autofill code is added.
• Partner with the Autofill Coverage Dataset engineers and the US Autofill team on defect triage and coordinated releases.

WHAT YOU BRING

• Overall Experience: 6-10 years in Software Engineering in Test, test automation, or performance engineering roles.
• Test Architecture: Proven track record designing and owning automated test harnesses and benchmarking infrastructure from the ground up, not just writing test cases against an existing framework.
• Browser Performance Internals: Working knowledge of browser rendering and performance APIs (Long Tasks, Long Animation Frames, User Timing, the main-thread/compositor split), and genuine comfort reasoning about why a given metric does or doesn't reflect what it appears to.
• Automation & Protocol Tooling: Experience building test harnesses with Playwright (or an equivalent) and working with an out-of-page debugging protocol (e.g., Chrome DevTools Protocol) for lab-only attribution.
• Statistical Literacy: Comfort choosing the right summary statistic for a dataset's actual shape and understanding what can and cannot be composed across an aggregation boundary.
• Data Modeling: Experience designing schemas/types for time-series or event data where correct join keys and clock alignment matter to the correctness of the analysis, not just its convenience.
• Technical Stack: Strong TypeScript; familiarity with browser extension architecture is a plus, since these benchmarks instrument the autofill content-script codebase directly.
• Observability Tooling: Experience publishing metrics to and alerting from a platform like Datadog.
• Open Source & Communication: Excellent written communication and a strong passion for open-source software, developer tooling, and modern internet security.

Why Join Us

This is an opportunity to play a foundational role in shaping our India Engineering COE from the ground up. Our user community loves us and we love them. Come to work each day with a sense of purpose as we bring a more secure internet experience to everyone from our friends and family to the world’s largest organizations.

Become an expert. You’ll get immersed in the prominent technology markets of security and open source software. You'll work alongside talented global teams, solve meaningful engineering challenges, influence how we scale our engineering organization & contribute to products used by customers around the world.

If you're passionate about building exceptional teams, enabling engineering excellence, and making a lasting impact within a fast-growing engineering company, we'd love to hear from you.

Equal Opportunity Employer

We are committed to fostering an inclusive workplace where everyone can do their best work. We welcome applicants from all backgrounds and make employment decisions based on skills, experience, qualifications, and business needs.