CHANGELOG-developer.next.asciidoc

// Use these for links to issue and pulls. Note issues and pulls redirect one to
// each other on Github, so don't worry too much on using the right prefix.
:issue: https://github.com/elastic/beats/issues/
:pull: https://github.com/elastic/beats/pull/

This changelog is intended for community Beat developers. It covers the major
breaking changes to the internal APIs in the official Beats and changes related
to developing a Beat like code generators or `fields.yml`. Only the major
changes will be covered in this changelog that are expected to affect community
developers. Each breaking change added here should have an explanation on how
other Beats should be migrated.

Note: This changelog documents the current changes which are not yet present in
an actual release.

=== Beats version HEAD
https://github.com/elastic/beats/compare/v7.0.0-rc2..main[Check the HEAD diff]

The list below covers the major changes between 7.0.0-rc2 and main only.

==== Breaking changes

- Move Fields from package libbeat/common to libbeat/mapping. {pull}11198[11198]
- For "metricbeat style" generated custom beats, the mage target `GoTestIntegration` has changed to `GoIntegTest` and `GoTestUnit` has changed to `GoUnitTest`. {pull}13341[13341]
- Build docker and kubernetes features only on supported platforms. {pull}13509[13509]
- Need to register new processors to be used in the JS processor in their `init` functions. {pull}13509[13509]
- The custom beat generator now uses mage instead of python, `mage GenerateCustomBeat` can be used to create a new beat, and `mage vendorUpdate` to update the vendored libbeat in a custom beat. {pull}13610[13610]
- Altered all remaining uses of mapval to use the renamed and enhanced version: https://github.com/elastic/go-lookslike[go-lookslike] instead, which is a separate project. The mapval tree is now gone. {pull}14165[14165]
- Move light modules to OSS. {pull}14369[14369]
- Deprecate test flags, `generate` and `update_expected`, in favor of `data`. {pull}15292[15292]
- Python 3 is required now to run python tests and tools. {pull}14798[14798]
- The type `memqueue.Broker` is no longer exported; instead of `memqueue.NewBroker`, call `memqueue.NewQueue` (which provides the same public interface). {pull}16667[16667]
- The disk spool types `spool.Spool` and `spool.Settings` have been renamed to the internal types `spool.diskSpool` and `spool.settings`. {pull}16693[16693]
- `queue.Eventer` has been renamed to `queue.ACKListener` {pull}16691[16691]
- Require logger as first parameter for `outputs.transport.transport#ProxyDialer` and `outputs.elasticsearch.client#BulkReadItemStatus`. {pull}16761[16761]
- The `libbeat/outputs/transport` package has been moved to `libbeat/common/transport`. {pull}16734[16734]
- The `libbeat/outputs/tls.go` file has been removed. All exported symbols in that file (`libbeat/outputs.*`) are now available as `libbeat/common/tlscommon.*`. {pull}16734[16734]
- The newly generated Beats are using go modules to manage dependencies. {pull}16288[16288]
- Extract Elasticsearch client logic from `outputs/elasticsearch` package into new `esclientleg` package. {pull}16150[16150]
- Rename `queue.BufferConfig.Events` to `queue.BufferConfig.MaxEvents`. {pull}17622[17622]
- Remove `queue.Feature` and replace `queue.RegisterType` with `queue.RegisterQueueType`. {pull}17666[17666]
- Introduce APM libbeat instrumentation. `Publish` method on `Client` interface now takes a Context as first argument. {pull}17938[17938]
- The way configuration files are generated has changed to make it easier to customize parts
  of the config without requiring changes to libbeat config templates. Generation is now
  fully based on Go text/template and no longer uses file concatenation to generate the config.
  Your magefile.go will require a change to adapt the devtool API. See the pull request for
  more details. {pull}18148[18148]
- The Elasticsearch client settings expect the API key to be raw (not base64-encoded). {issue}18939[18939] {pull}18945[18945]
- `management.ConfigManager` has been renamed to `management.Manager`. {pull}19114[19114]
- `UpdateStatus` has been added to the `management.Manager` interface. {pull}19114[19114]
- Remove `common.MapStrPointer` parameter from `cfgfile.Runnerfactory` interface. {pull}19135[19135]
- Replace `ACKCount`, `ACKEvents`, and `ACKLastEvent` callbacks with `ACKHandler` and interface in `beat.ClientConfig`. {pull}19632[19632]
- Remove global ACK handler support via `SetACKHandler` from publisher pipeline. {pull}19632[19632]
- Make implementing `Close` required for `reader.Reader` interfaces. {pull}20455[20455]
- Remove `NumCPU` as clients should update the CPU count on the fly in case of config changes in a VM. {pull}23154[23154]
- Remove Metricbeat EventFetcher and EventsFetcher interface. Use the reporter interface instead. {pull}25093[25093]
- Update Darwin build image to a debian 10 base that increases the MacOS SDK and minimum supported version used in build to 10.14. {issue}24193[24193]
- Removed the `common.Float` type. {issue}28279[28279] {pull}28280[28280] {pull}28376[28376]
- Removed Beat generators. {pull}28816[28816]
- libbeat.logp package forces ECS compliant logs. Logs are JSON formatted. Options to enable ECS/JSON have been removed. {issue}15544[15544] {pull}28573[28573]
- Removed deprecated disk spool from Beats. Use disk queue instead. {pull}28869[28869]
- Wildcard fields no longer have a default ignore_above setting of 1024. {issue}30096[30096] {pull}30668[30668]
- Remove `common.MapStr` and use `mapstr.M` from `github.com/elastic/elastic-agent-libs` instead. {pull}31420[31420]
- Remove `queue.Consumer`. Queues can now be read via a `Get` call directly on the queue object. {pull}31502[31502]
- The `queue.Batch` API now provides access to individual events instead of an array. {pull}31699[31699]
- Rename `queue.Batch.ACK()` to `queue.Batch.Done()`. {pull}31903[31903]
- `queue.ACKListener` has been removed. Queue configurations now accept an explicit callback function for ACK handling. {pull}35078[35078]
- Split split httpmon out of x-pack/filebeat/input/internal/httplog. {pull}36385[36385]
- Beats publishing pipeline does not propagate the close signal to its clients any more. It's responsibility of the user to close the pipeline client. {issue}38197[38197] {pull}38556[38556]
- Debug log entries from the acker (`stateful ack ...` or `stateless ack ...`) removed. {pull}39672[39672]
- Rename x-pack/filebeat websocket input to streaming. {issue}40264[40264] {pull}40421[40421]
- Journald input now calls `journalctl` instead of using `github.com/coreos/go-systemd/v22@v22.5.0/sdjournal`, the CGO dependency has been removed from Filebeat {pull}40061[40061]
- System module events now contain `input.type: systemlogs` instead of `input.type: log` when harvesting log files, however the ingest pipeline sets it back to the original input (log or journald). {pull}41246[41246]
- The system-logs input is removed because it's not used anymore {pull}42328[42328]

==== Bugfixes

- Handle the starting of namespace and node watchers for metadata enrichment according to `add_resource_metadata` configuration.{pull}38762[38762]
- Fix multiple metricbeat instances reporting same metrics when using autodiscover with provider kubernetes, and ensure leader elector is always running in autodiscover mode.{pull}38471[38471]
- Fix how Prometheus histograms are calculated when percentiles are provide.{pull}36537[36537]
- Stop using `mage:import` in community beats. This was ignoring the vendorized beats directory for some mage targets, using the code available in GOPATH, this causes inconsistencies and compilation problems if the version of the code in the GOPATH is different to the vendored one. Use of `mage:import` will continue to be unsupported in custom beats till beats is migrated to go modules, or mage supports vendored dependencies. {issue}13998[13998] {pull}14162[14162]
- Metricbeat module builders call host parser only once when instantiating light modules. {pull}20149[20149]
- Fix export dashboard command when running against Elastic Cloud hosted Kibana. {pull}22746[22746]
- Remove `event.dataset` (ECS) annotion from `libbeat.logp`. {issue}27404[27404]
- Errors should be thrown as errors. Metricsets inside Metricbeat will now throw errors as the `error` log level. {pull}27804[27804]
- Avoid panicking in `add_fields` processor when input event.Fields is a nil map. {pull}28219[28219]
- Drop event batch when get HTTP status 413 from Elasticsearch to avoid infinite loop {issue}14350[14350] {pull}29368[29368]
- Allow to use metricbeat for named mssql instances. {issue}24076[24076] {pull}30859[30859]
- Setting DEV=true when running `mage build` now correctly generates binaries without optimisations and with debug symbols {pull}31955[31955]
- The beat.cgroup.memory.mem.usage.bytes metric is now a gauge {issue}31582[31582] {pull}32652[32652]
- Fix the integration testcase docker port mapping for sql and oracle modules {pull}34221[34221]
- Fix the ingest pipeline for mysql slowlog to parse schema name with dash {pull}34371[34372]
- Fix the multiple host support for mongodb module {pull}34624[34624]
- Skip HTTPJSON flakey test. {issue}34929[34929] {pull}35138[35138]
- Fix ingest pipeline for panw module to parse url scheme correctly {pull}35757[35757]
- Renamed an httpjson input metric to follow naming conventions. `httpjson_interval_pages_total` was renamed to `httpjson_interval_pages` because the `_total` suffix is reserved for counters. {issue}35933[35933] {pull}36169[36169]
- Fixed some race conditions in tests {pull}36185[36185]
- Fix Stringer implementation of fingerprint processor {issue}35174[35174]
- Re-enable HTTPJSON fixed flakey test. {issue}34929[34929] {pull}36525[36525]
- Make winlogbeat/sys/wineventlog follow the unsafe.Pointer rules. {pull}36650[36650]
- Cleaned up documentation errors & fixed a minor bug in Filebeat Azure blob storage input. {pull}36714[36714]
- Fix copy arguments for strict aligned architectures. {pull}36976[36976]
- Fix panic when more than 32767 pipeline clients are active. {issue}38197[38197] {pull}38556[38556]
- Skip flakey metrics test on windows in filebeat httpjson input. {issue}39676[39676] {pull}39678[39678]
- Fix flakey test on Windows 2022 in packetbeat/route. {issue}39698[39698] {pull}39822[39822]
- Fix bug in minimum length for request trace logging. {pull}39834[39834]
- Close connections properly in Filbeat's HTTPJSON input. {pull}39790[39790]
- Add the Offset property to libbeat/reader.Message to store the total number of bytes read and discarded before generating the message. This enables inputs to accurately determine how much data has been read up to the message, using Message.Bytes + Message.Offset. {pull}39873[39873] {issue}39653[39653]
- AWS CloudWatch Metrics record previous endTime to use for next collection period and change log.logger from cloudwatch to aws.cloudwatch. {pull}40870[40870]
- Fix flaky test in cel and httpjson inputs of filebeat. {issue}40503[40503] {pull}41358[41358]
- Fix documentation and implementation of raw message handling in Filebeat http_endpoint by removing it. {pull}41498[41498]
- Fix flaky test in filebeat Okta entity analytics provider. {issue}42059[42059] {pull}42123[42123]
- Fix IIS module logging errors in case application pool PDH counter is not found. {pull}42274[42274]
- Removed direct dependency on Azure/go-autorest/autorest/adal, which is deprecated. {issue}41463[41463] {pull}42959[42959]

==== Added

- Update watchers to be shared between metricsets in Kubernetes module. {pull}37332[37332]
- Add new metricset in Kubernetes module, `state_namespace`. {pull}36406[36406]
- Add configuration for APM instrumentation and expose the tracer trough the Beat object. {pull}17938[17938]
- Make the behavior of clientWorker and netClientWorker consistent when error is returned from publisher pipeline
- Metricset generator generates beta modules by default now. {pull}10657[10657]
- The `beat.Event` accessor methods now support `@metadata` keys. {pull}10761[10761]
- Assertion for documented fields in tests fails if any of the fields in the tested event is documented as an alias. {pull}10921[10921]
- Support for Logger in the Metricset base instance. {pull}11106[11106]
- Filebeat modules can now use ingest pipelines in YAML format. {pull}11209[11209]
- Prometheus helper for metricbeat contains now `Namespace` field for `prometheus.MetricsMappings` {pull}11424[11424]
- Update Jinja2 version to 2.10.1. {pull}11817[11817]
- Reduce idxmgmt.Supporter interface and rework export commands to reuse logic. {pull}11777[11777],{pull}12065[12065],{pull}12067[12067],{pull}12160[12160]
- Update urllib3 version to 1.24.2 {pull}11930[11930]
- Add libbeat/common/cleanup package. {pull}12134[12134]
- New helper to check for leaked goroutines on tests. {pull}12106[12106]
- Only Load minimal template if no fields are provided. {pull}12103[12103]
- Add new option `IgnoreAllErrors` to `libbeat.common.schema` for skipping fields that failed while converting. {pull}12089[12089]
- Deprecate setup cmds for `template` and `ilm-policy`. Add new setup cmd for `index-management`. {pull}12132[12132]
- Use the go-lookslike library for testing in heartbeat. Eventually the mapval package will be replaced with it. {pull}12540[12540]
- New ReporterV2 interfaces that can receive a context on `Fetch(ctx, reporter)`, or `Run(ctx, reporter)`. {pull}11981[11981]
- Generate configuration from `mage` for all Beats. {pull}12618[12618]
- Add ClientFactory to TCP input source to add SplitFunc/NetworkFuncs per client. {pull}8543[8543]
- Introduce beat.OutputChooses publisher mode. {pull}12996[12996]
- Ensure that beat.Processor, beat.ProcessorList, and processors.ProcessorList are compatible and can be composed more easily. {pull}12996[12996]
- Add support to close beat.Client via beat.CloseRef (a subset of context.Context). {pull}13031[13031]
- Add checks for types and formats used in fields definitions in `fields.yml` files. {pull}13188[13188]
- Makefile included in generator copies files from beats repository using `git archive` instead of cp. {pull}13193[13193]
- Strip debug symbols from binaries to reduce binary sizes. {issue}12768[12768]
- Compare event by event in `testadata` framework to avoid sorting problems {pull}13747[13747]
- Added a `default_field` option to fields in fields.yml to offer a way to exclude fields from the default_field list. {issue}14262[14262] {pull}14341[14341]
- `supported-versions.yml` can be used in metricbeat python system tests to obtain the build args for docker compose builds. {pull}14520[14520]
- Fix dropped errors in the tests for the metricbeat Azure module. {pull}13773[13773]
- New mage target for Functionbeat: generate pkg folder to make manager easier. {pull}15580[15880]
- Add support for MODULE environment variable in `mage goIntegTest` in metricbeat to run integration tests for a single module. {pull}17147[17147]
- Add support for a `TEST_TAGS` environment variable to add tags for tests selection following go build tags semantics, this environment variable is used by mage test targets to add build tags. Python tests can also be tagged with a decorator (`@beat.tag('sometag')`). {pull}16937[16937] {pull}17075[17075]
- Add fields validation for histogram subfields. {pull}17759[17759]
- Add IP* fields to `fields.yml` generator script in Filebeat. {issue}17998[17998] {pull}18256[18256]
- Events intended for the Elasticsearch output can now take an `op_type` metadata field of type events.OpType or string to indicate the `op_type` to use for bulk indexing. {pull}12606[12606]
- Remove vendor folder from repository. {pull}18655[18655]
- Added SQL helper that can be used from any Metricbeat module {pull}18955[18955]
- Update Go version to 1.14.4. {pull}19753[19753]
- Update Go version to 1.14.7. {pull}20508[20508]
- Add packaging for docker image based on UBI minimal 8. {pull}20576[20576]
- Make the mage binary used by the build process in the docker container to be statically compiled. {pull}20827[20827]
- Add Pensando distributed firewall module. {pull}21063[21063]
- Update ecszap to v0.3.0 for using ECS 1.6.0 in logs {pull}22267[22267]
- Add support for customized monitoring API. {pull}22605[22605]
- Update Go version to 1.15.7. {pull}22495[22495]
- Update Go version to 1.15.8. {pull}23955[23955]
- Update Go version to 1.15.9. {pull}24442[24442]
- Update Go version to 1.15.10. {pull}24606[24606]
- Update Go version to 1.15.12. {pull}25629[25629]
- Update Go version to 1.16.4. {issue}25346[25346] {pull}25671[25671]
- Add sorting to array fields for generated data files (*-generated.json) {pull}25320[25320]
- Update to go-concert 0.2.0 {pull}27162[27162]
- Update Go version to 1.16.5. {issue}26182[26182] {pull}26186[26186]
- Introduce `libbeat/beat.Beat.OutputConfigReloader` {pull}28048[28048]
- Update Go version to 1.17.1. {pull}27543[27543]
- Whitelist `GCP_*` environment variables in dev tools {pull}28364[28364]
- Add support for `credentials_json` in `gcp` module, all metricsets {pull}29584[29584]
- Add gcp firestore metricset. {pull}29918[29918]
- Added TESTING_FILEBEAT_FILEPATTERN option for filebeat module pytests {pull}30103[30103]
- Improve tests files with shorter statements. {pull}35667[35667]
- Add gcp dataproc metricset. {pull}30008[30008]
- Add Github action for linting
- Add regex support for drop_fields processor.
- Improve compatibility and reduce flakyness of Python tests {pull}31588[31588]
- Added `.python-version` file {pull}32323[32323]
- Use `T.TempDir` to create temporary test directory {pull}33082[33082]
- Add an option to disable event normalization when creating a `beat.Client`. {pull}33657[33657]
- Add the file path of the instance lock on the error when it's is already locked {pull}33788[33788]
- Add DropFields processor to js API {pull}33458[33458]
- Add support for different folders when testing data {pull}34467[34467]
- Add logging of metric registration in inputmon. {pull}35647[35647]
- Add Okta API package for entity analytics. {pull}35478[35478]
- Add benchmarking to HTTPJSON input testing. {pull}35138[35138]
- Allow non-AWS endpoints for testing Filebeat awss3 input. {issue}35496[35496] {pull}35520[35520]
- Add AUTH (username) and SSL/TLS support for Redis module {pull}35240[35240]
- Pin PyYAML version to 5.3.1 to avoid CI errors temporarily {pull}36091[36091]
- Skip dependabot updates for github.com/elastic/mito. {pull}36158[36158]
- Add device handling to Okta API package for entity analytics. {pull}35980[35980]
- Make Filebeat HTTPJSON input process responses sequentially. {pull}36493[36493]
- Add initial infrastructure for a caching enrichment processor. {pull}36619[36619]
- Add file-backed cache for cache enrichment processor. {pull}36686[36686] {pull}36696[36696]
- Elide retryable HTTP client construction in Filebeat HTTPJSON and CEL inputs if not needed. {pull}36916[36916]
- Allow assignment of packetbeat protocols to interfaces. {issue}36574[36564] {pull}36852[36852]
- Add Active Directory entity collector for Filebeat entity analytics. {pull}37854[37854]
- Make logs for empty and small files less noisy when using fingerprint file identity in filestream. {pull}38421[38421]
- Improve robustness and error reporting from packetbeat default route testing. {pull}39757[39757]
- Move x-pack/filebeat/input/salesforce jwt import to v5. {pull}39823[39823]
- Drop x-pack/filebeat/input dependency on github.com/lestrrat-go/jwx/v2. {pull}39968[39968]
- Added `ignore_empty_values` flag in `decode_cef` Filebeat processor. {pull}40268[40268]
- Bump version of elastic/toutoumomoma to remove internal forks of stdlib debug packages. {pull}40325[40325]
- Refactor x-pack/filebeat/input/websocket for generalisation. {pull}40308[40308]
- Add a configuration option for TCP/UDP network type. {issue}40407[40407] {pull}40623[40623]
- Added debug logging to parquet reader in x-pack/libbeat/reader. {pull}40651[40651]
- Added filebeat debug histograms for s3 object size and events per processed s3 object. {pull}40775[40775]
- Simplified GCS input state checkpoint calculation logic. {issue}40878[40878] {pull}40937[40937]
- Simplified Azure Blob Storage input state checkpoint calculation logic. {issue}40674[40674] {pull}40936[40936]
- Add field redaction package. {pull}40997[40997]
- Add support for marked redaction to x-pack/filebeat/input/internal/private {pull}41212[41212]
- Add support for collecting Okta role and factor data for users with filebeat entityanalytics input. {pull}41044[41044]
- Add CEL input program evaluation coverage collection support. {pull}41884[41884]
- Fix test framework and python tests saving config file with wrong permission. {issue}42779[42779]
- Tests reading testdata config file now set flag `strict.perms=false`. They use the new `testflag` package helper function MustSetStrictPermsFalse. {issue}42779[42779]
- Add a metric registry to Filebeat's v2.Context. This should be the registry
used by inputs to publish metrics. This registry is then passed to the pipeline
client by the input/v2. When receiving a non-nil metrics registry the pipeline
client will aggregate metrics by input on this registry. It'll collect
`events_pipeline_total`, `events_pipeline_filtered_total`,
`events_pipeline_published_total`.{pull}42618[42618] {issue}42761[42761]
- Add new API to libbeat/monitoring/inputmon. The API allows to register and
unregister input metrics without relaying on the global 'dataset' namespace.{pull}42618[42618] {issue}42761[42761]
- Added exponential backoff retry logic for vSphere client logout operation in metricbeat. {issue}43449[43449]

==== Deprecated

- Deprecated the `common.Float` type. {issue}28279[28279] {pull}28280[28280]
- Deprecate Beat generators. {pull}28814[28814]
- Remove garbled PE executable test from auditbeat FIM module testing. {issue}35705[35705] {pull}35724[35724]