Add AIX support

[mostlyjason]

Describe the enhancement:
Currently, Filebeat does not support AIX but we've had several enhancement requests from customers to support it. AIX is not a common OS these days, but it does exist in great numbers in banking and insurance institutions that are highly regulated.

Additional details:
One customer has already created a version of Filebeat for AIX in version 5.5.1 but builds started to fail in 6.0. http://www.bullfreeware.com/pkg?id=4650

Go has gained support for AIX rather recently (minimum AIX 7.2), but some of our dependencies are not compiling on AIX. For example, Docker lacks support for AIX. You can see the conversation on Discuss here https://discuss.elastic.co/t/attempting-to-compile-filebeat-in-aix/143378/4 and https://discuss.elastic.co/t/disable-docker-support/178228. A fix has already been merged to use Docker only on supported platforms #13509. There are additional dependencies that need support for AIX as well.

[Helflym]

Hi,
I'm a member behind the Bullfreeware website and I'm currently actively porting Filebeat, Metricbeat and Auditbeat for AIX 7.2. Filebeat is already in good shape and I'll soon start pushing a few patches to introduce AIX to the beats software. Moreover, a rpm of version 7.5.1 should be able at the same time on our website, for those who are interested in.

About dependencies, as you said a fix has been merged to disable Docker and K8s for platforms which don't support it. A few adjustments have to be made in order to fully disable the build of Docker packages (especially with tests), but it's no longer a needed dependency for AIX.
The other dependencies (for Filebeat, Metricbeat and Auditbeat) that I'm aware of are:

• github.com/elastic/gosigar: In progress Add AIX support  gosigar#133
• github.com/elastic/go-sysinfo: In progress all: add AIX provider go-sysinfo#77
• github.com/fsnotify/fsnotify: In progress Add polling fallback fsnotify/fsnotify#326
• github.com/elastic/go-txfile: Merged Add AIX support go-txfile#44. Wait for gofrs to vendor
• github.com/gofrs/flock: In progress Add AIX support with fcntl gofrs/flock#40
• github.com/lib/pq: Merged Add AIX support lib/pq#929. Vendor possible
• github.com/insomniacslk/dhcp : Merged interfaces: add support for AIX insomniacslk/dhcp#352. Vendor possible
• github.com/mattn/go-isatty: Merged Add AIX support mattn/go-isatty#35, Vendor possible

I'll keep posting updates on this issue.
Note, maybe the title can be changed to "Add AIX support" as Filebeat isn't the only one targeted.

Edit 2020/04/01: Update dependencies

[mostlyjason]

@Helflym Awesome thanks so much this is an amazing effort and I'm excited to see all the progress you've made. Definitely keep us up to date here as you make more progress. I also updated the title as you suggested.

I heard separately that metricbeat on AIX will require more effort due to the OS API being different. Are you working on that as well?

[Helflym]

I heard separately that metricbeat on AIX will require more effort due to the OS API being different. Are you working on that as well?

Yes of course. For the moment, it seems that porting metrics dependencies (gosigar, go-sysinfo, etc) is enough to have a working metricbeat. Maybe the data won't be fully accurate and some fields will be missing. But it will be enough for a first metricbeat, and I will have time to improve them if needed.

[Helflym]

A quick update to say that I've released a first beta version of filebeat, metricbeat and auditbeat under our website: http://www.bullfreeware.com/?searching=true&package=beat&from=&to=&libraries=false&exact=false&version=5. Any feedback about configuration and/or modules would be much appreciated.

I've started to push some patches for dependencies upstream but almost all of them are stuck right now. So, we'll have to wait to have AIX on the beats' upstream.
elastic/gosigar#133
elastic/go-sysinfo#77
shirou/gopsutil#834
fsnotify/fsnotify#9

[mostlyjason]

Thanks for this effort @Helflym

[adammike]

@Helflym are these the apache licensed filebeat/auditbeat code?

[Helflym]

@adammike I think yes. The maintainers will know more about this than me.

[sanandece]

@Helflym I want to try filbeat on AIX 7.2 to ship application logs to elastic and looking if there is a documented procedure. If there is one, can you point me to it please?

[Helflym]

@sanandece for now, the only available option to install filebeat is to use our RPM (http://www.bullfreeware.com/pkg?id=5615). We are planning to upstream the changes asap.

[Helflym]

I've updated the dependencies in my first post and added which ones have the AIX support merged in their upstream and which ones are still in progress. I've also added if they can be vendored here.

[mikemadden42]

For this effort are we targeting a set of POWER versions? It looks like Go only supports AIX 7.2.

From this article, it looks like AIX 7.2 needs POWER7 or newer CPUs. Are we going to target POWER7 & POWER8?

[Helflym]

Golang toolchain only AIX 7.2 and Power8+ is supported.
With gccgo, we might be able to build it for Power7, there will be a bit of work to do.

[inxonic]

I heard separately that metricbeat on AIX will require more effort due to the OS API being different. Are you working on that as well?

@mostlyjason @Helflym Just wanted to note that I put some effort into getting AIX specific metrics into Beats through the perfstat API a while ago. As of now that's far from complete regarding the set of metrics that could eventually be fetched. However it works for a few and the implementation could serve as an example for others. That's the code: https://github.com/WuerthIT/perfstatbeat

[Helflym]

@inxonic I've already done the work in order to have metricbeat working: elastic/gosigar#133 and elastic/go-sysinfo#77. However, same as you, I've only added the "mandatory" metrics for beats, some more can be added.
I'm also seen that you're building it with gccgo, did you try with the Golang Toolchain ?

[fearful-symmetry]

Just a brief update -- There's still a couple open PRs on elastic's end. We're waiting until we have an AIX dev environment until we merge them.

[serak]

can i use Linux metricbeat binary on AIX because on download page of https://www.elastic.co/downloads/beats/metricbeat i am not seeing any AIX image?

[Helflym]

@serak no, you can't for multiple various, the first one being AIX having a different executable format than Linux.
However, we've ported version 7.5.1 for AIX 7.2, it can be found there: http://www.bullfreeware.com/pkg?id=5616.

[fearful-symmetry]

Hey folks, brief update for anyone tracking. Elastic finally has access to an AIX test environment, so I can start testing/developing support in metricbeat.

[fearful-symmetry]

Another update for y'all. The changes to go-sysinfo and gosigar have been merged. Working on beats itself now, starting with metricbeat.

[Helflym]

@fearful-symmetry I've checked the patches I've done when starting the port earlier this year. Most of them we'll be useless for you (the one dealing with either dependencies or magefiles) but I guess a few are still worth it.
I'm giving them as is. I mean to say that they are 6 months old so they might not be accurate anymore. Sadly I don't have time to update them right now...

0001-auditbeat-module-file_integrity-port-for-AIX.txt
0001-libbeat-processors-add_host_metadata-add-AIX.txt
0001-libbeat-processors-add_process_metadata-fix-TestBadP.txt
0001-metricbeat-add-system-metric-support-for-AIX.txt

[serak]

@serak no, you can't for multiple various, the first one being AIX having a different executable format than Linux.
However, we've ported version 7.5.1 for AIX 7.2, it can be found there: http://www.bullfreeware.com/pkg?id=5616.

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

user/# rpm -qR filebeat
/usr/bin/env
libc.a(shr_64.o)
libperfstat.a(shr_64.o)
libpthreads.a(shr_xpg5_64.o)
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1

appt1test:/opt/freeware/bin# prtconf
System Model: IBM,9117-MMC
Machine Serial Number: 064AC87
Processor Type: PowerPC_POWER7
Processor Implementation Mode: POWER 7
Processor Version: PV_7_Compat
Number Of Processors: 6
Processor Clock Speed: 3304 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 4 app1tst
Memory Size: 35840 MB
Good Memory Size: 35840 MB
Platform Firmware level: AM770_123
Firmware Version: IBM,AM770_123
Console Login: enable
Auto Restart: true
Full Core: false

[Helflym]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

[serak]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

Thank you for your quick response. the error message was useless. i have a couple of Power9 servers i will try on those. how is this issue going for a native support for aix? what is missing right now?

[Helflym]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

Thank you for your quick response. the error message was useless. i have a couple of Power9 servers i will try on those. how is this issue going for a native support for aix? what is missing right now?

You're right, I should have displayed a warning when installing filebeat packages on Power7, as we do for Golang.
The problem is not with AIX itself, the Go toolchain on Power is made to work only on Power8 and later. It also impacts Linux. There was some discussions about supporting Power7 but it was refused, I don't remember why and where though.

[rajeshrgithub]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

Thank you for your quick response. the error message was useless. i have a couple of Power9 servers i will try on those. how is this issue going for a native support for aix? what is missing right now?

You're right, I should have displayed a warning when installing filebeat packages on Power7, as we do for Golang.
The problem is not with AIX itself, the Go toolchain on Power is made to work only on Power8 and later. It also impacts Linux. There was some discussions about supporting Power7 but it was refused, I don't remember why and where though.

In this case, Do I need to install Go toolchain separately on Power 8 or already built in function. What are the pre-requisite required inorder to start up metricbeat on Power 8.

[Helflym]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

Thank you for your quick response. the error message was useless. i have a couple of Power9 servers i will try on those. how is this issue going for a native support for aix? what is missing right now?

You're right, I should have displayed a warning when installing filebeat packages on Power7, as we do for Golang.
The problem is not with AIX itself, the Go toolchain on Power is made to work only on Power8 and later. It also impacts Linux. There was some discussions about supporting Power7 but it was refused, I don't remember why and where though.

In this case, Do I need to install Go toolchain separately on Power 8 or already built in function. What are the pre-requisite required inorder to start up metricbeat on Power 8.

Normally, you don't need anything special, not even the Go toolchain. The metricbeat provided by BullFreeware is static.

[rajeshrgithub]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

Thank you for your quick response. the error message was useless. i have a couple of Power9 servers i will try on those. how is this issue going for a native support for aix? what is missing right now?

You're right, I should have displayed a warning when installing filebeat packages on Power7, as we do for Golang.
The problem is not with AIX itself, the Go toolchain on Power is made to work only on Power8 and later. It also impacts Linux. There was some discussions about supporting Power7 but it was refused, I don't remember why and where though.

In this case, Do I need to install Go toolchain separately on Power 8 or already built in function. What are the pre-requisite required inorder to start up metricbeat on Power 8.

Normally, you don't need anything special, not even the Go toolchain. The metricbeat provided by BullFreeware is static.

Thanks for your response. Only metricbeat-7.5.2-1.aix7.2.ppc.rpm package is enough from AIX client side right. I will be testing this package on power 8 server and metericbeat.yml should be fine to onboard in ELK dashboard.

[rajeshrgithub]

Successfully installed meteicbeat on power 8, how to start the daemon and configure.

[serak]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

Thank you for your quick response. the error message was useless. i have a couple of Power9 servers i will try on those. how is this issue going for a native support for aix? what is missing right now?

You're right, I should have displayed a warning when installing filebeat packages on Power7, as we do for Golang.
The problem is not with AIX itself, the Go toolchain on Power is made to work only on Power8 and later. It also impacts Linux. There was some discussions about supporting Power7 but it was refused, I don't remember why and where though.

In this case, Do I need to install Go toolchain separately on Power 8 or already built in function. What are the pre-requisite required inorder to start up metricbeat on Power 8.

Not required unless you want to compile the source code :). just install the rpm and run it using "nohup /opt/freeware/bin/filebeat &" or just /opt/freeware/bin/filebeat -e to see the logs and close of session disconnect (for testing )

[cheokpl20]

Successfully installed the filebeat-7.5.1-1beta.aix7.2.ppc.rpm in AIX 7.2 and encountered the Segmentation fault(coredump) problem when startup the filebeat using the command /opt/freeware/share/filebeat/bin/filebeat. Need help urgently. Thanks

$ rpm -qa
expect-5.42.1-3.ppc
tk-8.4.7-3.ppc
libgcc10-10.3.0-6.ppc
libgcc-10-1.ppc
libgo8-8.3.0-6.ppc
df1000e21410f103-00012000040025700030-1.noarch
AIX-rpm-7.2.5.1-5.ppc
bash-5.1.8-1.ppc
tcl-8.4.7-3.ppc
libgo10-10.3.0-6.ppc
libgo-10-1.ppc
libgcc8-8.3.0-6.ppc
pci.1014034a.51-19512b00-1.noarch
libiconv-1.16-5.ppc
filebeat-7.5.1-1beta.ppc

[Helflym]

First, could you confirm that you're running on a Power8 or higher ? This filebeat version from BullFreeware isn't working for Power7.

Secondly, could try to run gdb with the command ? It should be able to show which functions is causing the Segfault.

[cheokpl20]

Yes. The system is running in Power 9. We are not able to analyse the core dump using the gdb because the gdb only available for AIX 6.1 and cannot run in AIX 7.2

[Helflym]

Yes. The system is running in Power 9. We are not able to analyse the core dump using the gdb because the gdb only available for AIX 6.1 and cannot run in AIX 7.2

AIX ensures backward compatibility. Thus any sofware available in AIX 6.1 will work as is in AIX 7.2. So you can run gdb without any trouble.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[serak]

has anybody tried cross compile the source code mage -compile=beats_magefile -goos=aix -goarch=ppc64 i am getting the following error

# github.com/elastic/go-sysinfo/providers/aix
..\..\..\go\pkg\mod\github.com\elastic\[email protected]\providers\aix\os_aix.go:36:23: undefined: getKernelVersion
Error: error compiling magefiles

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[LBoraz]

would be nice to be able to install beats or elastic agent on AIX

[FreerideTheWorld]

Time for Elastic to take a look at this project again. At a minimum Metricbeat and Filebeat should be available in a version 8.x format. AIX + Beats questions are plentiful on the discussion forums with no uptodate solutions. This thread have been open since 2020 with seemingly no progress or other easy solutions.

[FreerideTheWorld]

Still no movement on this? Unless I'm missing a thread somewhere. Surely there has to be enough of a user base that would like Elastic Agent 8.x (Metricbeat/Filebeat) on AIX even if it isn't a common OS since "it does exist in great numbers in banking and insurance institutions that are highly regulated." Those two industries aren't short on money. You can also add large government agencies to that list so it's about time Elastic puts some effort into this so users don't have to use third party tools to efficiently get their metrics and logs.

[fred-maussion]

Cross referencing the issue on AIX elastic-agent support issue on the elastic-agent repository for visibility.