Buf

Appearance

Protovalidate quickstart ​

This quickstart steps through using Protovalidate in Protobuf projects with the Buf CLI:

  1. Adding Protovalidate rules to schemas.
  2. Using CEL to add domain-specific validation logic.
  3. Enabling server-side validation.

Download the code (optional) ​

If you'd like to code along in Go, TypeScript/JavaScript, Java, or Python, complete the following steps. If you're only here for a quick tour, feel free to skip ahead.

  1. Install the Buf CLI. If you already have, run buf --version to verify that you're using at least 1.54.0.

  2. Have git and your choice of go, Node.js, Java 17+, or Python 3.7+ installed.

  3. Clone the buf-examples repository:

    sh
    git clone https://github.com/bufbuild/buf-examples.git

  4. Open a terminal to the repository and navigate to the protovalidate/quickstart-go/start, protovalidate/quickstart-es/start, protovalidate/quickstart-java/start, or protovalidate/quickstart-python/start directory.

Each language's quickstart code contains Buf CLI configuration files (buf.yaml, buf.gen.yaml), a simple weather_service.proto, and an idiomatic unit test.

Add Protovalidate to schemas ​

Depend on Protovalidate ​

Published publicly on the Buf Schema Registry, the Protovalidate module provides the Protobuf extensions, options, and messages powering validation.

Add it as a dependency in buf.yaml:

buf.yaml

yaml
version: v2
modules:
  - path: proto
deps:
  - buf.build/bufbuild/protovalidate:v0.11.1
lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE

Next, update dependencies. You may see a warning that Protovalidate hasn't yet been used. That's fine.

sh
buf dep update
WARN    Module buf.build/bufbuild/protovalidate is declared in your buf.yaml deps but is unused...

If you're using Go or Java, update managed mode options in buf.gen.yaml:

buf.gen.yaml

yaml
version: v2
inputs:
  - directory: proto
plugins:
  - remote: buf.build/protocolbuffers/go:v1.36.5
    out: gen
    opt:
      - paths=source_relative
managed:
  enabled: true
  override:
    - file_option: go_package_prefix
      value: github.com/bufbuild/buf-examples/protovalidate/quickstart-go/start/gen
# Don't modify any file option or field option for protovalidate. Without
# this, generated Go will fail to compile.
disable:
  - file_option: go_package
    module: buf.build/bufbuild/protovalidate

Add rules to a message ​

To add rules to a message, you'll first import Protovalidate and then add Protovalidate annotations.

Make the following changes to proto/bufbuild/weather/v1/weather_service.proto to add rules to a GetWeatherRequest message. (Java note: this directory is relative to src/main.)

proto/bufbuild/weather/v1/weather_service.proto

protobuf
syntax = "proto3";

package bufbuild.weather.v1;

import "buf/validate/validate.proto"; 
import "google/protobuf/timestamp.proto";

// GetWeatherRequest is a request for weather at a point on Earth.
message GetWeatherRequest {
  // latitude must be between -90 and 90, inclusive, to be valid. Use of a
  // float allows precision to about one square meter.
  float latitude = 1; 
  float latitude = 1 [
   (buf.validate.field).float.gte = -90,
   (buf.validate.field).float.lte = 90
  ];

  // longitude must be between -180 and 180, inclusive, to be valid. Use of a
  // float allows precision to about one square meter.
  float longitude = 2; 
  float longitude = 2 [
   (buf.validate.field).float.gte = -180,
   (buf.validate.field).float.lte = 180
  ];

  // forecast_date for the weather request. It must be within the next
  // three days.
  google.protobuf.Timestamp forecast_date = 3;
}

Run this code

You can run this example in the Protovalidate playground, a miniature IDE where Protovalidate rules can be tested against sample payloads.

Lint your changes ​

It's possible to add rules to a message that compile but cause unexpected results or exceptions at runtime. If the prior example is changed to require latitude but to also skip its validation when unpopulated, it contains a logical contradiction:

A logical contradiction within a message

protobuf
message GetWeatherRequest {
  float latitude = 1 [
    (buf.validate.field).ignore = IGNORE_IF_UNPOPULATED,
    (buf.validate.field).required = true,
    (buf.validate.field).float.gte = -90,
    (buf.validate.field).float.lte = 90
  ];
}

The Buf CLI's lint command identifies these and other problems, like invalid CEL expressions, with its PROTOVALIDATE rule :

Buf lint errors for the PROTOVALIDATE rule

sh
buf lint
proto/bufbuild/weather/v1/weather_service.proto:29:5:Field "latitude" has both
(buf.validate.field).required and (buf.validate.field).ignore=IGNORE_IF_UNPOPULATED.
A field cannot be empty if it is required.

We recommend using buf lint any time you're editing schemas, as well in GitHub Actions or other CI/CD tools.

Build the module ​

Now that you've added Protovalidate as a dependency, updated your schema with rules, and validated changes with buf lint, your module should build with no errors:

sh
buf build

Generate code ​

Protovalidate doesn't introduce any new code generation plugins because its rules are compiled as part of your service and message descriptors — buf generate works without any changes.

Run it to include your new rules in the GetWeatherRequest descriptor:

sh
buf generate

To learn more about generating code with the Buf CLI, read the code generation overview.

Add business logic with CEL ​

If Protovalidate only provided logical validations on known types, such as maximum and minimum values or verifying required fields were provided, it'd be an incomplete library. Real world validation rules are often more complicated:

  1. A BuyMovieTicketsRequest request must be for a showtime in the future but no more than two weeks in the future.
  2. A SaveBlogEntryRequest must have a status of DRAFT, PUBLISHED, or ARCHIVED.
  3. An AddProductToInventoryRequest must have a serial number starting with a constant prefix and matching a complicated regular expression.

Protovalidate can meet all of these requirements because all Protovalidate rules are defined in Common Expression Language (CEL). CEL is a lightweight, high-performance expression language that allows expressions like this.first_flight_duration + this.second_flight_duration < duration('48h') to evaluate consistently across languages.

Adding a CEL-based rule to a field is straightforward. Instead of a providing a static value, you provide a unique identifier (id), an error message, and a CEL expression. Building on the prior GetWeatherRequest example, add a custom rule stating that users must ask for weather forecasts within the next 72 hours:

proto/bufbuild/weather/v1/weather_service.proto

protobuf
syntax = "proto3";

package bufbuild.weather.v1;

import "buf/validate/validate.proto";
import "google/protobuf/timestamp.proto";

// GetWeatherRequest is a request for weather at a point on Earth.
message GetWeatherRequest {
  // latitude must be between -90 and 90, inclusive, to be valid. Use of a
  // float allows precision to about one square meter.
  float latitude = 1 [
    (buf.validate.field).float.gte = -90,
    (buf.validate.field).float.lte = 90
  ];
  // longitude must be between -180 and 180, inclusive, to be valid. Use of a
  // float allows precision to about one square meter.
  float longitude = 2 [
    (buf.validate.field).float.gte = -180,
    (buf.validate.field).float.lte = 180
  ];

  // forecast_date for the weather request. It must be within the next
  // three days.
  google.protobuf.Timestamp forecast_date = 3; 
  google.protobuf.Timestamp forecast_date = 3 [(buf.validate.field).cel = {
     id: "forecast_date.within_72_hours"
     message: "Forecast date must be in the next 72 hours."
     expression: "this >= now && this <= now + duration('72h')"
  }];
}

Remember to recompile and regenerate code:

sh
buf generate

Run this code

You can run this example in the Protovalidate playground, a miniature IDE where Protovalidate rules can be tested against sample payloads.

Run validation ​

All Protovalidate languages provide an idiomatic API for validating a Protobuf message.

In the final code exercise, you'll use it directly, checking enforcement of GetWeatherRequest's validation rules.

  1. Make sure you've navigated to protovalidate/quickstart-go/start within the buf-examples repository.

  2. Install Protovalidate using go get.

    sh
    go get buf.build/go/protovalidate@v0.12.0

  3. Run weather/weather_test.go with go test. It should fail — it expects invalid latitudes and longitudes to be rejected, but you haven't yet added any validation.

    sh
    go test ./weather
--- FAIL: TestRequests (0.00s)
    --- FAIL: TestRequests/latitude_too_low (0.00s)
        weather_test.go:65:
                Error Trace:    /Users/janedoe/dev/bufbuild/buf-examples/protovalidate/quickstart-go/start/weather/weather_test.go:65
                Error:          An error is expected but got nil.
                Test:           TestRequests/latitude_too_low
    --- FAIL: TestRequests/latitude_too_high (0.00s)
        weather_test.go:65:
                Error Trace:    /Users/janedoe/dev/bufbuild/buf-examples/protovalidate/quickstart-go/start/weather/weather_test.go:65
                Error:          An error is expected but got nil.
                Test:           TestRequests/latitude_too_high
FAIL
FAIL    github.com/bufbuild/buf-examples/protovalidate/quickstart-go/start/weather  0.244s
FAIL

  4. Open weather/weather.go. Update the validateWeather function to return the result of protovalidate.Validate():

    weather/weather.go

    go
    package weather

import (
    weatherv1 "github.com/bufbuild/buf-examples/protovalidate/quickstart-go/start/gen/bufbuild/weather/v1"
    "github.com/bufbuild/protovalidate-go"
)

func validateWeather(_ *weatherv1.GetWeatherRequest) error { 
      // TODO: validate the request
      return nil
} 
func validateWeather(req *weatherv1.GetWeatherRequest) error { 
      return protovalidate.Validate(req) 
}

  5. Run go test. Now that you've added validation, all tests should pass.

    sh
    go test ./weather

You've now walked through the basic steps for using Protovalidate: adding it as a dependency, annotating your schemas with rules, and validating Protobuf messages.

Validate API requests ​

One of Protovalidate's most common use cases is for validating requests made to RPC APIs. Though it's possible to use the above examples to add a validation request at the start of every request handler, it's not efficient. Instead, use Protovalidate within a ConnectRPC or gRPC interceptor, providing global input validation.

Open-source Protovalidate interceptors are available for Connect Go and gRPC-Go. In the quickstarts for specific languages and gRPC frameworks, you'll also find example interceptors for Java, and Python.

Adding these interceptors is no different from configuring any other RPC interceptor:

go
// Create the validation interceptor provided by connectrpc.com/validate.
interceptor, err := validate.NewInterceptor()
if err != nil {
    log.Fatal(err)
}

// Include the interceptor when adding handlers.
path, handler := weatherv1connect.NewWeatherServiceHandler(
    weatherServer,
    connect.WithInterceptors(interceptor),
)

For a deep dive into using Protovalidate for RPC APIs, explore one of the Protovalidate integration quickstarts:

Validate Kafka messages ​

In traditional Kafka, brokers are simple data pipes — they have no understanding of what data traverses them. Though this simplicity helped Kafka gain ubiquity, most data sent through Kafka topics is structured and should follow a schema.

Using Bufstream — the Kafka-compatible message queue built for the data lakehouse era — you can add Protovalidate rule enforcement to broker-side schema awareness. With a Bufstream broker already using the Buf Schema Registry's Confluent Schema Registry support, enabling Protovalidate is a two-line configuration change within data_enforcement:

Bufstream Configuration YAML

yaml
data_enforcement:
  produce:
    - topics: { all: true }
      values:
        on_parse_error: REJECT_BATCH
        validation:
          on_error: REJECT_BATCH

For a deep dive into using Protovalidate with Bufstream, follow the Protovalidate in Kafka quickstart.

Next steps ​

Read on to learn more about enabling schema-first validation with Protovalidate: