How to configure Elasticsearch, Filebeat and Kibana to view WSO2 Carbon logs

This blog will explain the most basic steps one should follow to configure Elasticsearch, Filebeat and Kibana to view WSO2 product logs.

Pre-requisites

I have written this document assuming that we are using the below product versions.

Download the below versions of Elasticsearch, filebeat and Kibana.
Elasticsearch - 5.1.1
Filebeat - 5.1.1
Kibana - 5.1.1

How to configure Filebeat

1. Download Filebeat to the server where you Carbon Product is running.
2. You can install it in any of the methods mentioned at [1].
3. Then, open up the filebeat.yml file and change the file path mentioned under filebeat.prospectors.

filebeat.prospectors:
- input_type: log
  paths:
    - /home/ubuntu/wso2esb-4.9.0/repository/logs/wso2carbon.log


4. Configure the output.elasticsearch and point to the server where you are running Elasticsearch.

output.elasticsearch:
  hosts: ["192.168.52.99:9200"]
 
5. If you are using a template other that what's being used by default, you can change the configuration as below.

output.elasticsearch:
  hosts: ["192.168.52.99:9200"]
  template.name: "filebeat"
  template.path: "filebeat.template-es2x.json"
  template.overwrite: false 



6. Once the above configuration are done, start your Filebeat server using the options given at [2].

Configuring ElasticSearch

1. For better performance, it is requested to use Elasticsearch on JDK 1.8. Hence, as the first step, make sure you install JDK 1.8.0 on your machine before continuing with the rest of the steps mentioned here.

2. Install Elasticsearch using the below command

sudo dpkg -i elasticsearch-5.1.1.deb


3. For the most basic scenario, you only need to configure the host by specifying the IP of the node that Elasticsearch is running on.

network.host: 192.168.52.99

4. Now start the ElasticSearch server.

sudo service elasticsearch start

Viewing the logs from Kibana

1. Extract Kibana to a preferred location.

2. Open the kibana.yml file and point to your Elasticsearch server.

elasticsearch.url: "http://192.168.52.99:9200"

3. Access the Kibana server from the URL http://localhost:5601 and you can view the WSO2 carbon logs.

[1]  - https://www.elastic.co/guide/en/beats/filebeat/5.x/filebeat-installation.html
[2] - https://www.elastic.co/guide/en/beats/filebeat/5.x/filebeat-starting.html

How to access an ActiveMQ queue from WSO2 ESB which is secured with a username/password

By default, a queue in ActiveMQ can be accessed without providing any credentials. However, in real world scenarios, you will have to deal with secured queues. So in this blog, I will explain how we can enable security for ActiveMQ and what configurations are required to be done in WSO2 ESB.

Pr-requisites - Enable the JMS transport for WSO2 ESB as explained in [1].

Step 1 - Secure the ActiveMQ instance with credentials.

To do this, add the below configuration to the activemq.xml under the <broker> tag and start the server.

<plugins>
    <simpleAuthenticationPlugin anonymousAccessAllowed="true">
        <users>
            <authenticationUser username="system" password="system" groups="users,admins"/>
            <authenticationUser username="admin" password="admin" groups="users,admins"/>
            <authenticationUser username="user" password="user" groups="users"/>
            <authenticationUser username="guest" password="guest" groups="guests"/>
        </users>
    </simpleAuthenticationPlugin>
</plugins>

Step 2 - Enable the JMS Listener configuration and configure it as shown below.

    <!--Uncomment this and configure as appropriate for JMS transport support, after setting up your JMS environment (e.g. ActiveMQ)-->
    <transportReceiver name="jms" class="org.apache.axis2.transport.jms.JMSListener">
        <parameter name="myTopicConnectionFactory" locked="false">
                <parameter name="java.naming.factory.initial" locked="false">org.apache.activemq.jndi.ActiveMQInitialContextFactory</parameter>
                <parameter name="java.naming.provider.url" locked="false">tcp://localhost:61616</parameter>
                <parameter name="java.naming.security.principal" locked="false">admin</parameter>
                <parameter name="java.naming.security.credentials" locked="false">admin</parameter>
                <parameter locked="false" name="transport.jms.UserName">admin</parameter>
                <parameter locked="false" name="transport.jms.Password">admin</parameter>
                <parameter name="transport.jms.ConnectionFactoryJNDIName" locked="false">TopicConnectionFactory</parameter>
                <parameter name="transport.jms.ConnectionFactoryType" locked="false">topic</parameter>
        </parameter>

        <parameter name="myQueueConnectionFactory" locked="false">
                <parameter name="java.naming.factory.initial" locked="false">org.apache.activemq.jndi.ActiveMQInitialContextFactory</parameter>
                <parameter name="java.naming.provider.url" locked="false">tcp://localhost:61616</parameter>
                <parameter name="java.naming.security.principal" locked="false">admin</parameter>
                <parameter name="java.naming.security.credentials" locked="false">admin</parameter>
                <parameter locked="false" name="transport.jms.UserName">admin</parameter>
                <parameter locked="false" name="transport.jms.Password">admin</parameter>
                <parameter name="transport.jms.ConnectionFactoryJNDIName" locked="false">QueueConnectionFactory</parameter>
                <parameter name="transport.jms.ConnectionFactoryType" locked="false">queue</parameter>
        </parameter>

        <parameter name="default" locked="false">
                <parameter name="java.naming.factory.initial" locked="false">org.apache.activemq.jndi.ActiveMQInitialContextFactory</parameter>
                <parameter name="java.naming.provider.url" locked="false">tcp://localhost:61616</parameter>
                <parameter name="java.naming.security.principal" locked="false">admin</parameter>
                <parameter name="java.naming.security.credentials" locked="false">admin</parameter>
                <parameter locked="false" name="transport.jms.UserName">admin</parameter>
                <parameter locked="false" name="transport.jms.Password">admin</parameter>
                <parameter name="transport.jms.ConnectionFactoryJNDIName" locked="false">QueueConnectionFactory</parameter>
                <parameter name="transport.jms.ConnectionFactoryType" locked="false">queue</parameter>
        </parameter>
    </transportReceiver>

Step 3 - Create a Proxy service to listen to a JMS queue in ActiveMQ.

Once the ESB server is started, create the below Proxy service and let it listen to the queue generated in ActiveMQ.


   <proxy name="StockQuoteProxy1" transports="jms" startOnLoad="true">
      <target>
         <endpoint>
            <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
         </endpoint>
         <inSequence>
            <property name="OUT_ONLY" value="true"/>
         </inSequence>
         <outSequence>
            <send/>
         </outSequence>
      </target>
      <publishWSDL uri="file:repository/samples/resources/proxy/sample_proxy_1.wsdl"/>
      <parameter name="transport.jms.ContentType">
         <rules>
            <jmsProperty>contentType</jmsProperty>
            <default>application/xml</default>
         </rules>
      </parameter>
   </proxy>
Once the above proxy service is deployed, send a request to the queue and observe how the message is processed and send to the backend. You can use the sample available in [2] to test this scenario out.

If you are sending a JMS request you can use the username and the password in the URL as shown below.
ant stockquote -Dmode=placeorder -Dtrpurl="jms:/StockQuoteProxy1?transport.jms.DestinationType=queue&transport.jms.ContentTypeProperty=contentType&java.naming.provider.url=tcp://localhost:61616&java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&transport.jms.UserName="admin"&transport.jms.Password="admin"&transport.jms.ConnectionFactoryType=queue&transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory"


[1] - https://docs.wso2.com/display/ESB490/Configure+with+ActiveMQ
[2] - https://docs.wso2.com/display/ESB490/Sample+250%3A+Introduction+to+Switching+Transports